Kubernetes pod eviction and traffic draining for to be evicted pod

Question

I'm trying understand to Kuberentes pod eviction behaviour and when it decides to terminate a pod how it manages that no request should be in process state when it is going to terminate a pod.

It is based on a recent troubleshooting of incident where 3 requests resulted in 502 bad gateway on the ingress controller Kong.

During the analysis it was found out that at the exact same time K8s terminated a pod because it was hitting ephemeral storage limits set on one of the container of the pod. In addition to this on linkerd container logged that connection was reset by peer and proxy was failed to process the request. This connection reset translated to 502 bad gateway for ingress controller kong.

I understand by improving the limits and controlling the number bytes container writes on disk I can avoid eviction by kubernetes to a great extent.

I'm Interested before terminating a pod does kuberentes makes sure that that no traffic is load balanced on to be terminated pod. Also what are the best practices to fine tune Kubernetes pod eviction.

Answer 1

setup a readiness Probe in the pod spec, something like this

readinessProbe:
  httpGet:
    path: /
    port: 443

readiness probes specifically inform service routing decisions

Answer 2

The pod eviction can happen because of resources outages like CPU, ram, and storage. You can prevent your cluster from the eviction of pod by declaring the resource quotas. You can define resource quota over namespace but before that, you have to calculate workload usage and create a resource limit on provided request quota. You have three types of quotas available.

1. Compute Quotas
2. Storage Quotas
3. Object Count Quotas

There are a few more ways to quotas on the kubernetes cluster.

https://kubernetes.io/docs/concepts/policy/resource-quotas/

Answer 3

To answer your two questions:

I'm Interested before terminating a pod does kuberentes makes sure that that no traffic is load balanced on to be terminated pod.

If you have more than 1 pod running then yes, it will not send traffic to the failed/terminating pod.

Also what are the best practices to fine tune Kubernetes pod eviction.

The official Kubernetes documentation explains the Pod lifetime cycle in great detail along with suggestions on how to fine tune. While liveness probes and readiness probes can help determine the health of a pod, they will not help in the case of eviction.

Eviction occurs due to a lack of resources (your case), or if the Node is unavailable (e.g., spot instance eviction, Node maintenance). By using something like a ReplicaSet or Deployment you can specify how many pods you want to have running and can fine tune where these run. For instance a DaemonSet will by default deploy to every node. Architecting this way will ensure that if a node goes down, or a pod fails to run due to lack of resources, it will have other nodes available or will be rescheduled to launch on another node.