I have a cluster running on AWS EKS, v1.20.7-eks-d88609
I have a Pod and a NodePort Service to expose it to the outside.
Some clients will connect to it through a TCP connection which will be kept alive.
However, I was trying to configure a graceful termination of the connection when the Pod was terminated, by capturing the SIGTERM and sending a final message in the TCP connection telling the clients to disconnect.
Problem is these final TCP packets (sent after the SIGTERM is detected) never arrive at the destination.
I have run a tcpdump inside the Pod and confirmed that the packet is being sent correctly: https://imgur.com/Bhy6fL1
But it does not arrive at the destination (where I had another tcpdump to check this)
TCP even tries to retransmit the packet since an ack for it didn't arrive, but none of them are able to reach the destination.
I was not able to reproduce this in a local Minikube cluster running same version of Kubernetes (1.20.7). Things worked well in this local cluster.
I'm not sure the best way to debug this in the AWS EKS cluster. It seems to be some issue caused by something inside Kubernetes, since the app tries to send the packet.