1

I've inherited an app with K8s services with the following config

spec:
  ...
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate

For most of these light-weight services, we run a single instance of a pod on GCP. We're expanding into blue-green deploys for some services, but the majority are still run this way.

When we update the GCR image to a new version, the service becomes temporarily unavailable (maybe 10-30 seconds). I'm assuming this is because K8s is killing the old pod while it brings up the new pod. Is there anyway to configure K8s to actually bring up a new pod, direct new traffic to it, and only then shutdown the old pod? This would seemingly allow for a deployment with no downtime, and not rely on having two pods running all the time (saving cost). All docs I can find typically start with N pods so they don't have this issue...


UPDATE

So the issue is not related to maxsurge (changing it had no effect). Actually I realized what's happening is that the pod is getting traffic routed to it before the app is actually completely ready. These are spring apps, so they take like 30 seconds to start, and the service disruption we're seeing is occurring in the 30s window while the app is booting and the pod has already sent traffic to it. So I guess what I really am after is as follows

how can k8s be configured to bring up a second pod, wait until it passes startup probe/healthcheck, and only then have the first pod route traffic to it and self-terminate?

1
  • 1
    Hi Adam! Could you please replace the screenshot with a code snippet? This will make it easier to find this question. – Bruce Becker Feb 25 at 3:49
3

Is there anyway to configure K8s to actually bring up a new pod, direct new traffic to it, and only then shutdown the old pod?

This is the default behavior. MaxSurge is number of additional pods that runs during upgrade.

What you also have to check is how the load balancer is handling traffic, does it have capability to detect that you have deployed a new Pod, e.g. is it Kubernetes aware? See Container native load balancing on Google Cloud, it looks like you should have a ReadinessProbe on your Deployment.

Actually I realized what's happening is that the pod is getting traffic routed to it before the app is actually completely ready.

This is the default behavior, if you define a ReadinessProbe for your Deployment. As you write, you can also add a StartupProbe but the ReadinessProbe is most important.

4
  • Ah awesome. Let me try it out! Thank you for pointing me in the right direction. I'll accept the answer after I see if that solves it, and digest the other link you've sent me... – Adam Hughes Feb 23 at 18:22
  • Please see updated question if still interestd - maxsurge was not the issue – Adam Hughes Feb 27 at 0:00
  • I have updated, but I had already answered that you need a ReadinessProbe - so no really change. – Jonas Feb 27 at 6:41
  • Ok thanks - sorry I didn't pay close enough attention! – Adam Hughes Feb 28 at 15:55
0

The difficulties we encountered with a single pod rolling update are already explained here:

https://stackoverflow.com/questions/46369100/kubernetes-rolling-update-killing-off-old-pod-without-bringing-up-new-one

The solution to us was multifaceted. First, we needed to set MaxUnavailable = 0 and MaxSurge=2. We then needed to setup a persistent volume claim, so that the pods could share disk. If each pod has its own disk, then the pods would just hang, as the second pod is never "available" to take over.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.