Can we do zero-downtime kubernetes deployment with a single pod?

Question

I've inherited an app with K8s services with the following config

spec:
  ...
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate

For most of these light-weight services, we run a single instance of a pod on GCP. We're expanding into blue-green deploys for some services, but the majority are still run this way.

When we update the GCR image to a new version, the service becomes temporarily unavailable (maybe 10-30 seconds). I'm assuming this is because K8s is killing the old pod while it brings up the new pod. Is there anyway to configure K8s to actually bring up a new pod, direct new traffic to it, and only then shutdown the old pod? This would seemingly allow for a deployment with no downtime, and not rely on having two pods running all the time (saving cost). All docs I can find typically start with N pods so they don't have this issue...

---

UPDATE

So the issue is not related to maxsurge (changing it had no effect). Actually I realized what's happening is that the pod is getting traffic routed to it before the app is actually completely ready. These are spring apps, so they take like 30 seconds to start, and the service disruption we're seeing is occurring in the 30s window while the app is booting and the pod has already sent traffic to it. So I guess what I really am after is as follows

how can k8s be configured to bring up a second pod, wait until it passes startup probe/healthcheck, and only then have the first pod route traffic to it and self-terminate?

Answer 1

Is there anyway to configure K8s to actually bring up a new pod, direct new traffic to it, and only then shutdown the old pod?

This is the default behavior. MaxSurge is number of additional pods that runs during upgrade.

What you also have to check is how the load balancer is handling traffic, does it have capability to detect that you have deployed a new Pod, e.g. is it Kubernetes aware? See Container native load balancing on Google Cloud, it looks like you should have a ReadinessProbe on your Deployment.

Actually I realized what's happening is that the pod is getting traffic routed to it before the app is actually completely ready.

This is the default behavior, if you define a ReadinessProbe for your Deployment. As you write, you can also add a StartupProbe but the ReadinessProbe is most important.

Answer 2

The difficulties we encountered with a single pod rolling update are already explained here:

https://stackoverflow.com/questions/46369100/kubernetes-rolling-update-killing-off-old-pod-without-bringing-up-new-one

The solution to us was multifaceted. First, we needed to set MaxUnavailable = 0 and MaxSurge=2. We then needed to setup a persistent volume claim, so that the pods could share disk. If each pod has its own disk, then the pods would just hang, as the second pod is never "available" to take over.

Answer 3

I would suggest you go with ReadinessProbe. So, with this, your application can say that "I am ready to receive the traffic" then the current running pod is ready to be terminated.

Note that it is different to LivenessProbe where it means "I am alive", "but not yet ready to serve the traffic".