I'm trying understand to Kuberentes pod eviction behaviour and when it decides to terminate a pod how it manages that no request should be in process state when it is going to terminate a pod.
It is based on a recent troubleshooting of incident where 3 requests resulted in 502 bad gateway on the ingress controller Kong.
During the analysis it was found out that at the exact same time K8s terminated a pod because it was hitting ephemeral storage limits set on one of the container of the pod. In addition to this on linkerd container logged that connection was reset by peer and proxy was failed to process the request. This connection reset translated to 502 bad gateway for ingress controller kong.
I understand by improving the limits and controlling the number bytes container writes on disk I can avoid eviction by kubernetes to a great extent.
I'm Interested before terminating a pod does kuberentes makes sure that that no traffic is load balanced on to be terminated pod. Also what are the best practices to fine tune Kubernetes pod eviction.