how to control pod termination speed in kubernetes

Question

I have a k8s deployment that I update by doing kubectl apply -f spec.yml. I use the default update strategy.

Whenever I update, new pods get created and the old ones eventually get deleted.

This is what kubectl get pods looks like a few seconds after calling apply:

NAME                      READY     STATUS        RESTARTS   AGE
server-243710380-7s7g0    1/1       Terminating   0          9m
server-243710380-ffzv1    1/1       Terminating   0          9m
server-243710380-gfs3h    1/1       Terminating   0          9m
server-3561703442-24dxm   1/1       Running       0          13s
server-3561703442-7jlds   1/1       Running       0          8s
server-3561703442-x5xw1   1/1       Running       0          11s

As you can see, the ones in Terminating status hang around, typically for about 30 seconds, before really disappearing. This happens despite the new pods reaching running status within a few seconds.

How can I control the time the old pods remain in terminating status? Specifically, I want them to terminate as quickly as possible.

Answer 1

There is a default grace period for pod termination of 30 seconds. See: https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods.

You can customize the grace period setting terminationGracePeriodSeconds at the pod spec level.

Answer 2

Note : It is one of the Kubernetes best practices to allow termination with grace.

Kubernetes waits for a specified time called the termination grace period. By default, this is 30 seconds. It’s important to note that this happens in parallel to the preStop hook and the SIGTERM signal.

NOTE : Kubernetes does not wait for the preStop hook to finish.If your app finishes shutting down and exits before the terminationGracePeriod is done, Kubernetes moves to the next step immediately.

If your pod usually takes longer than 30 seconds to shut down, make sure you increase the grace period. You can do that by setting the terminationGracePeriodSeconds option in the Pod YAML.

It’s important that your application terminate gracefully so that there is minimal impact on the end user and the time-to-recovery is as fast as possible!

To change it to value lower or higher than default value of 30 seconds include the flag as below example.

Example :

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: nginx
  name: nginx
spec:
  containers:
  - image: nginx
    name: nginx
  terminationGracePeriodSeconds: 0

We can always make use of kubectl explain command to find supported options under spec.

For example : kubectl explain deployment.spec.template.spec

master $ kubectl explain deployment.spec.template.spec.terminationGracePeriodSeconds
KIND:     Deployment
VERSION:  apps/v1

FIELD:    terminationGracePeriodSeconds <integer>

DESCRIPTION:
     Optional duration in seconds the pod needs to terminate gracefully. May be
     decreased in delete request. Value must be non-negative integer. The value
     zero indicates delete immediately. If this value is nil, the default grace
     period will be used instead. The grace period is the duration in seconds
     after the processes running in the pod are sent a termination signal and
     the time when the processes are forcibly halted with a kill signal. Set
     this value longer than the expected cleanup time for your process. Defaults
     to 30 seconds.