2

I have an in house 5 node cluster running on bare-metal and I am using Calico. The cluster was working for 22 days but suddenly it stopped working. After investigating the problem I found out that the service to pod communication is broken while all the components are up and kubectl is working without a problem.

From within the cluster (component A) if I try to curl another component (bridge) with its IP it works:

$ curl -vvv http://10.4.130.184:9998
* Rebuilt URL to: http://10.4.130.184:9998/
*   Trying 10.4.130.184...
* TCP_NODELAY set
* Connected to 10.4.130.184 (10.4.130.184) port 9998 (#0)
> GET / HTTP/1.1
> Host: 10.4.130.184:9998
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Accept-Ranges: bytes
< Cache-Control: public, max-age=0
< Last-Modified: Mon, 08 Apr 2019 14:06:42 GMT
< ETag: W/"179-169fd45c550"
< Content-Type: text/html; charset=UTF-8
< Content-Length: 377
< Date: Wed, 23 Oct 2019 09:56:35 GMT
< Connection: keep-alive
< 
<!doctype html>
<html lang="en">
<head>
    <meta charset="utf-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />

    <title>Bridge</title>

    <meta content='width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=0' name='viewport' />
    <meta name="viewport" content="width=device-width" />
</head>
<body>
    <h1>Bridge</h1>
</body>

</html>
* Connection #0 to host 10.4.130.184 left intact

ns lookup for the service is also working (it resolve to the service IP):

$ nslookup bridge
Server:    10.5.0.10
Address 1: 10.5.0.10 kube-dns.kube-system.svc.k8s.local

Name:      bridge
Address 1: 10.5.160.50 bridge.170.svc.k8s.local

But the service to pod communication is broken and when I curl to the service name most of the times (60-70%) it stuck:

$ curl -vvv http://bridge:9998
* Rebuilt URL to: http://bridge:9998/
* Could not resolve host: bridge
* Closing connection 0
curl: (6) Could not resolve host: bridge

When I check the endpoints of the that service I can see that the IP of that pod is there:

$ kubectl get ep -n 170 bridge
NAME     ENDPOINTS                                               AGE
bridge   10.4.130.184:9226,10.4.130.184:9998,10.4.130.184:9226   11d

But as I said the curl (and any other method) that uses the service name is not working. And this is the service description:

$ kubectl describe svc -n 170 bridge
Name:              bridge
Namespace:         170
Labels:            io.kompose.service=bridge
Annotations:       Process: bridge
Selector:          io.kompose.service=bridge
Type:              ClusterIP
IP:                10.5.160.50
Port:              9998  9998/TCP
TargetPort:        9998/TCP
Endpoints:         10.4.130.184:9998
Port:              9226  9226/TCP
TargetPort:        9226/TCP
Endpoints:         10.4.130.184:9226
Port:              9226-udp  9226/UDP
TargetPort:        9226/UDP
Endpoints:         10.4.130.184:9226
Session Affinity:  None
Events:            <none>

This problem is not limited to just this component and it is like this for all the components.

I restarted the CoreDNS (deleted its pods) but it is still the same. I was facing this problem before and previous time I thought it is related to Weavenet that I was using and I needed the cluster so I tore down the cluster and rebuilt it with Calico but now I am sure that this is not related to CNI and it is something else.

Environment: - Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

  • Cloud provider or hardware configuration: This is a bare-metal cluster of 5 nodes, 1 master and 4 workers. All nodes are running Ubuntu 18.04 and they are connecting to the same subnet.

  • OS (e.g: cat /etc/os-release):

NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
  • Kernel (e.g. uname -a):
Linux serflex-argus-1 4.15.0-55-generic #60-Ubuntu SMP Tue Jul 2 18:22:20 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

  • Install tools: Kubeadm

  • Network plugin and version (if this is a network-related bug): Calico "cniVersion": "0.3.1"

Update

After deleting all the kube-proxy pods problem seemed to be solved but I still like to know what caused this problem. Btw I didn't see any error in the kube-proxy logs.

Update 2

The journalctl also doesn't show any strange activity, it just shows some logs when I deleted the pod of kube-proxy:

giant-7:~$ sudo journalctl --since "4 days ago" | grep kube-proxy

Oct 23 12:24:16 serflex-giant-7 kubelet[31145]: I1023 12:24:16.751108   31145 reconciler.go:181] operationExecutor.UnmountVolume started for volume "kube-proxy" (UniqueName: "kubernetes.io/configmap/02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea-kube-proxy") pod "02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea" (UID: "02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea")
Oct 23 12:24:16 serflex-giant-7 kubelet[31145]: I1023 12:24:16.751237   31145 reconciler.go:181] operationExecutor.UnmountVolume started for volume "kube-proxy-token-4t5tq" (UniqueName: "kubernetes.io/secret/02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea-kube-proxy-token-4t5tq") pod "02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea" (UID: "02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea")
Oct 23 12:24:16 serflex-giant-7 kubelet[31145]: W1023 12:24:16.765057   31145 empty_dir.go:421] Warning: Failed to clear quota on /var/lib/kubelet/pods/02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea/volumes/kubernetes.io~configmap/kube-proxy: ClearQuota called, but quotas disabled
Oct 23 12:24:16 serflex-giant-7 kubelet[31145]: I1023 12:24:16.782557   31145 operation_generator.go:831] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea-kube-proxy" (OuterVolumeSpecName: "kube-proxy") pod "02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea" (UID: "02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea"). InnerVolumeSpecName "kube-proxy". PluginName "kubernetes.io/configmap", VolumeGidValue ""
Oct 23 12:24:16 serflex-giant-7 kubelet[31145]: I1023 12:24:16.840793   31145 operation_generator.go:831] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea-kube-proxy-token-4t5tq" (OuterVolumeSpecName: "kube-proxy-token-4t5tq") pod "02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea" (UID: "02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea"). InnerVolumeSpecName "kube-proxy-token-4t5tq". PluginName "kubernetes.io/secret", VolumeGidValue ""
Oct 23 12:24:16 serflex-giant-7 kubelet[31145]: I1023 12:24:16.851656   31145 reconciler.go:301] Volume detached for volume "kube-proxy-token-4t5tq" (UniqueName: "kubernetes.io/secret/02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea-kube-proxy-token-4t5tq") on node "serflex-giant-7" DevicePath ""
Oct 23 12:24:16 serflex-giant-7 kubelet[31145]: I1023 12:24:16.851679   31145 reconciler.go:301] Volume detached for volume "kube-proxy" (UniqueName: "kubernetes.io/configmap/02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea-kube-proxy") on node "serflex-giant-7" DevicePath ""
Oct 23 12:24:25 serflex-giant-7 kubelet[31145]: I1023 12:24:25.973757   31145 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy" (UniqueName: "kubernetes.io/configmap/4e7f5d97-fd49-461b-ae38-6bc5e3f0462b-kube-proxy") pod "kube-proxy-qpj4h" (UID: "4e7f5d97-fd49-461b-ae38-6bc5e3f0462b")
Oct 23 12:24:25 serflex-giant-7 kubelet[31145]: I1023 12:24:25.973826   31145 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "lib-modules" (UniqueName: "kubernetes.io/host-path/4e7f5d97-fd49-461b-ae38-6bc5e3f0462b-lib-modules") pod "kube-proxy-qpj4h" (UID: "4e7f5d97-fd49-461b-ae38-6bc5e3f0462b")
Oct 23 12:24:25 serflex-giant-7 kubelet[31145]: I1023 12:24:25.973958   31145 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy-token-4t5tq" (UniqueName: "kubernetes.io/secret/4e7f5d97-fd49-461b-ae38-6bc5e3f0462b-kube-proxy-token-4t5tq") pod "kube-proxy-qpj4h" (UID: "4e7f5d97-fd49-461b-ae38-6bc5e3f0462b")
Oct 23 12:24:25 serflex-giant-7 kubelet[31145]: I1023 12:24:25.974027   31145 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "xtables-lock" (UniqueName: "kubernetes.io/host-path/4e7f5d97-fd49-461b-ae38-6bc5e3f0462b-xtables-lock") pod "kube-proxy-qpj4h" (UID: "4e7f5d97-fd49-461b-ae38-6bc5e3f0462b")
Oct 23 12:24:26 serflex-giant-7 systemd[1]: Started Kubernetes transient mount for /var/lib/kubelet/pods/4e7f5d97-fd49-461b-ae38-6bc5e3f0462b/volumes/kubernetes.io~secret/kube-proxy-token-4t5tq.
Oct 23 12:24:26 serflex-giant-7 kubelet[31145]: E1023 12:24:26.645571   31145 kuberuntime_manager.go:920] PodSandboxStatus of sandbox "ff4c1ba15b8c11a4fe86974286d81ebe4870d9e670226234fae6c7c21ce36c1d" for pod "kube-proxy-qpj4h_kube-system(4e7f5d97-fd49-461b-ae38-6bc5e3f0462b)" error: rpc error: code = Unknown desc = Error: No such container: ff4c1ba15b8c11a4fe86974286d81ebe4870d9e670226234fae6c7c21ce36c1d
Improve this question
6
  • Is there anything in /var/log/kube-proxy.log logs that would suggest kube-proxy had somehow failed. In addition, you can check iptables rules are being written by kube-proxy using the command: iptables-save | grep hostnames
    – Richard Slater
    Commented Oct 23, 2019 at 13:05
  • That log file doesn't exist: cat: /var/log/kube-proxy.log: No such file or directory and also the sudo iptables-save | grep hostnames returns nothing.
    – AVarf
    Commented Oct 23, 2019 at 13:36
  • Hmm... googling about a bit suggest that means you are on an OS that is using systemd in which case you need to use journalctl to view the logs (I think). If there are no iptables then I think that means kube-proxy is working in userspace, I'm going to struggle to help with this one as my Kubernetes experience is limited to non-SystemD and kernel mode kube-proxy. Does iptables-save return anything?
    – Richard Slater
    Commented Oct 23, 2019 at 14:32
  • I updated my question with the result of journalctl but I don't see anything strange. iptables-save returns pages of rules but when I grep for hostnames nothing is there.
    – AVarf
    Commented Oct 23, 2019 at 14:49
  • I tend to run through Debugging Services when I run into problems that can't trivially be explained from the logs; there is a section in there on using iptables-save to identify if rules are not being created by kube-proxy.
    – Richard Slater
    Commented Oct 23, 2019 at 14:55

1 Answer 1

Reset to default
1

Sounds like the problem has been resolved by deleting all of the kube-proxy pods, which is an aggressive solution but if it works then good work!

Best I can offer you is a collection of links about diagnosing Kubernetes problems:

Not a great answer I'm afraid, but we may have missed the opportunity to diagnose the issue.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.