0

Is it possible to tell kubernetes, that when performing a deployment of multiple pods that it should start only one at a time and only create the next pod if the previous has completely started up?

It would be useful in our usecase: Our application automatically migrates/upgrades the database when starting up. But when there are multiple pods starting up at the same time, it could happen that multiple pods try to upgrade the database at the same time which could corrupt the database.

But if kubernetes would wait with starting up the second pod until the first pod has fully started up, it would mean that the database has been correctly upgraded by then.

If kubernetes does not have such functionality, how would you handle such a case?

1

Use a combination of Readiness Probes and a proper deployment strategy.

  1. Configure your Deployment spec like so:
    spec:
      replicas: 4    # or as many as you need to have
      strategy:
        rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
    
    maxUnavailable = 0 means that there should always be a number of pods available equal to the replicas count. maxSurge = 1 means that kubernetes will create the new pods one at a time, and destroy the old ones one-at-a-time as well.
  2. Implement a readinessProbe which should return a successful response when the database migration has completed. You'd have to write the logic for that yourself, but if you do it right, kubernetes will create a pod with the new Deployment and wait for it to complete and only after that, evict 1 of the old ones.

PS: It's worth noting that this approach will cause your node to temporarily contain more pods than your replicas config implies, namely replicas + 1 for a short period of time while deploying. You need to make sure that your node can handle the little extra load untill the rollout is complete.

0

Here you could create a lock table inside your database, then the first pod would take the lock and then run the update. Other pods startup scripts would wait for the lock to be released before starting the application. A simpler solution is to run a pod inside a job for upgrading the database before launching the application.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.