How to recreate GCP Preemptible VMs behind LB before they expire?

Question

I created preemptible instances in a managed instance group behind LB, I specified the preemptible option in the instance template.

Because GCP Compute Engine always terminates preemptible instances after they run for 24 hours, I need a way to launch new preemptible VMs before they goes down.

Is GCP Compute Engine launch new preemptible VMs before the current ones goes down when target size specified by the autoscaler ?

Maybe need to replace the Preemptible VMs several hours before they going down,

I am not sure what is the best way to do it, maybe with termination notices.

Basically what I need is always to keep on Preemptible VMs behind the LB.

What is the best way to implement this?

Answer 1

Strictly you can't do this with a single preemptable virtual machine as you only get a 30 second grace period. While the majority of calls to the Google Compute Engine API would create a new device within 30 seconds, there are cases when this would fail due to network latency or temporary outage.

What you can do is create an Instance Group of Preemptible Virtual Machines with a minimum of two instances, the probability of GCE killing both at precisely the same time is quite low, and once GCE destroys one a new one will be created to bring the IG instance count back up to two.