TCP communication blocked during Pod termination in Kubernetes

Question

I have a cluster running on AWS EKS, v1.20.7-eks-d88609

I have a Pod and a NodePort Service to expose it to the outside.

Some clients will connect to it through a TCP connection which will be kept alive.

However, I was trying to configure a graceful termination of the connection when the Pod was terminated, by capturing the SIGTERM and sending a final message in the TCP connection telling the clients to disconnect.

Problem is these final TCP packets (sent after the SIGTERM is detected) never arrive at the destination.

I have run a tcpdump inside the Pod and confirmed that the packet is being sent correctly: https://i.stack.imgur.com/eW53z.jpg

But it does not arrive at the destination (where I had another tcpdump to check this)

TCP even tries to retransmit the packet since an ack for it didn't arrive, but none of them are able to reach the destination.

I was not able to reproduce this in a local Minikube cluster running same version of Kubernetes (1.20.7). Things worked well in this local cluster.

I'm not sure the best way to debug this in the AWS EKS cluster. It seems to be some issue caused by something inside Kubernetes, since the app tries to send the packet.

Answer 1

I've found the problem after some time, will add an answer here in case someone finds this in the future.

Basically, my problem was exactly what is described in https://github.com/aws/amazon-vpc-cni-k8s/issues/1531

I had Calico installed in my cluster and was using a version that had a bug which blocked the Pod networking during the termination phase.

This was the tutorial I followed when I installed Calico: https://docs.aws.amazon.com/eks/latest/userguide/calico.html

All I had to do to fix the problem was to uninstall Calico and install it again using a newer version (by following this same tutorial again)

---

Additional tip:

One of the things that helped me debug this problem was to make sure my apps weren't the ones to blame on this behavior.

I did this by running tcpdump in the Pod and forwarding its output to a local Wireshark for visualization. See this: https://downey.io/blog/kubernetes-ephemeral-debug-container-tcpdump/

After I found out my apps were sending traffic correctly, I knew the problem was something inside Kubernetes and started looking for possible culprits.