1

Really looking for some advice and help, I have run out of ideas. Basically everything was working fine, 1 master and 3 workers. One Worker node was turned off to increase CPU and RAM resources and once turned back on, PODs running on this worker were not able to reach any other PODs and also DNS resolving was not working, so no external network connection also was working, e.g. google.com etc. Here is a list of POD cidrs

$ kubectl get nodes -o json | jq -r '.items[].spec.podCIDR' | sort
Master - 10.244.0.0/24
Worker1 - 10.244.1.0/24
Worker2 - 10.244.2.0/24
Worker3 - 10.244.5.0/24

The one which was restarted and stopped working is Worker3. When running dnsutils container on this node and trying some debuging

root@dnsutils:/# nslookup kubernetes.default
;; connection timed out; no servers could be reached

Firewalld has been disabled and turned off. Also when trying to ping all the networks from Worker3, only 10.244.1.0 and 10.244.5.0 (itself) is pinging

root@dnsutils:/# ping 10.244.0.0
PING 10.244.0.0 (10.244.0.0) 56(84) bytes of data.
^C
--- 10.244.0.0 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1043ms

root@dnsutils:/# ping 10.244.1.0
PING 10.244.1.0 (10.244.1.0) 56(84) bytes of data.
64 bytes from 10.244.1.0: icmp_seq=1 ttl=63 time=0.721 ms
64 bytes from 10.244.1.0: icmp_seq=2 ttl=63 time=0.947 ms
^C
--- 10.244.1.0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1041ms
rtt min/avg/max/mdev = 0.721/0.834/0.947/0.113 ms
root@dnsutils:/# ping 10.244.2.0
PING 10.244.2.0 (10.244.2.0) 56(84) bytes of data.
^C
--- 10.244.2.0 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1024ms

root@dnsutils:/# ping 10.244.5.0
PING 10.244.5.0 (10.244.5.0) 56(84) bytes of data.
64 bytes from 10.244.5.0: icmp_seq=1 ttl=64 time=0.116 ms
64 bytes from 10.244.5.0: icmp_seq=2 ttl=64 time=0.142 ms
^C
--- 10.244.5.0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1038ms
rtt min/avg/max/mdev = 0.116/0.129/0.142/0.013 ms

I believe the issue is because it cant reach coredns service/pods, which both are currently on 10.244.0.35 and 10.244.0.36 IPs

NAME                                                                 READY   STATUS    RESTARTS         AGE     IP              NODE        NOMINATED NODE   READINESS GATES
calico-kube-controllers-7b564c7d85-rkc98                             1/1     Running   0                278d    10.244.0.30     Master       <none>           <none>
canal-5cffv                                                          2/2     Running   4 (378d ago)     630d    10.203.8.167    Worker3      <none>           <none>
canal-p822m                                                          2/2     Running   2 (474d ago)     630d    10.203.8.165    Master       <none>           <none>
canal-scdrs                                                          2/2     Running   4 (351d ago)     630d    10.203.40.166   Worker2      <none>           <none>
canal-xkx7f                                                          2/2     Running   0                5h16m   10.203.8.174    Worker4      <none>           <none>
coredns-cc4c6799f-4tjbd                                              1/1     Running   0                11h     10.244.0.36     Master       <none>           <none>
coredns-cc4c6799f-bkq8n                                              1/1     Running   0                11h     10.244.0.35     Master       <none>           <none>
etcd-Master                                                          1/1     Running   6 (474d ago)     630d    10.203.8.165    Master       <none>           <none>
kube-apiserver-Master                                                1/1     Running   6 (474d ago)     630d    10.203.8.165    Master       <none>           <none>
kube-controller-manager-Master                                       1/1     Running   10 (265d ago)    392d    10.203.8.165    Master       <none>           <none>
kube-proxy-9n4wc                                                     1/1     Running   1 (474d ago)     630d    10.203.8.165    Master       <none>           <none>
kube-proxy-rtccq                                                     1/1     Running   2 (351d ago)     630d    10.203.40.166   Worker2      <none>           <none>
kube-proxy-vtr55                                                     1/1     Running   0                5h16m   10.203.8.174    Worker4      <none>           <none>
kube-proxy-w9wkk                                                     1/1     Running   2 (378d ago)     630d    10.203.8.167    Worker3      <none>           <none>
kube-scheduler-Master                                                1/1     Running   326 (265d ago)   630d    10.203.8.165    Master       <none>           <none>
kube-state-metrics-8945855d-pvqqt                                    1/1     Running   0                139d    10.244.2.89     Worker3      <none>           <none>
metrics-server-7cf8b65d65-qpnzm                                      1/1     Running   2 (6h43m ago)    139d    10.244.2.83     Worker3      <none>           <none>

Running dnsutils on Worker2 for example

root@dnsutils2:/# nslookup kubernetes.default
Server:     10.96.0.10
Address:    10.96.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.96.0.1

root@dnsutils2:/# ping 10.244.0.0
PING 10.244.0.0 (10.244.0.0) 56(84) bytes of data.
64 bytes from 10.244.0.0: icmp_seq=1 ttl=63 time=0.787 ms
64 bytes from 10.244.0.0: icmp_seq=2 ttl=63 time=0.835 ms
^C
--- 10.244.0.0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1061ms
rtt min/avg/max/mdev = 0.787/0.811/0.835/0.024 ms
root@dnsutils2:/# ping 10.244.1.0
PING 10.244.1.0 (10.244.1.0) 56(84) bytes of data.
64 bytes from 10.244.1.0: icmp_seq=1 ttl=64 time=0.129 ms
64 bytes from 10.244.1.0: icmp_seq=2 ttl=64 time=0.152 ms
^C
--- 10.244.1.0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1021ms
rtt min/avg/max/mdev = 0.129/0.140/0.152/0.016 ms
root@dnsutils2:/# ping 10.244.2.0
PING 10.244.2.0 (10.244.2.0) 56(84) bytes of data.
64 bytes from 10.244.2.0: icmp_seq=1 ttl=63 time=1.03 ms
64 bytes from 10.244.2.0: icmp_seq=2 ttl=63 time=0.933 ms
^C
--- 10.244.2.0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.933/0.984/1.036/0.060 ms
root@dnsutils2:/# ping 10.244.5.0
PING 10.244.5.0 (10.244.5.0) 56(84) bytes of data.
64 bytes from 10.244.5.0: icmp_seq=1 ttl=63 time=0.787 ms
64 bytes from 10.244.5.0: icmp_seq=2 ttl=63 time=0.995 ms
^C
--- 10.244.5.0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1024ms
rtt min/avg/max/mdev = 0.787/0.891/0.995/0.104 ms

Cluster was initialized running below command

kubeadm init --apiserver-advertise-address=10.203.8.165 --pod-network-cidr=10.244.0.0/16

And for network cni plugin flannel-canal was used

kubectl apply -f https://projectcalico.docs.tigera.io/manifests/canal.yaml

I would really appreciate any help on where to start debugging this. Basically this Worker3 was working just fine before a restart and admins say that nothing was changed from network side or security groups etc. This is running on premises on VMware VMs.

I have also tried kubeadm reset and then iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X , rebooting the node and then kubeadm join but still the issue persists. No idea why this specific Worker3 wont ping anything besides 10.244.1.0 and 10.244.5.0 (itself)

I am confused about why wouldnt node be able to ping 10.244.2.0 network for example, but is able to ping 10.244.1.0 although there is iptables

-A FORWARD -s 10.244.0.0/16 -j ACCEPT
-A FORWARD -d 10.244.0.0/16 -j ACCEPT

and also routes

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.203.8.254    0.0.0.0         UG    100    0        0 ens192
10.203.8.0      0.0.0.0         255.255.255.0   U     100    0        0 ens192
10.244.0.0      10.244.0.0      255.255.255.0   UG    0      0        0 flannel.1
10.244.1.0      10.244.1.0      255.255.255.0   UG    0      0        0 flannel.1
10.244.2.0      10.244.2.0      255.255.255.0   UG    0      0        0 flannel.1
10.244.5.2      0.0.0.0         255.255.255.255 UH    0      0        0 cali952544557fa
10.244.5.3      0.0.0.0         255.255.255.255 UH    0      0        0 calic15b507754a
10.244.5.4      0.0.0.0         255.255.255.255 UH    0      0        0 calif987de568c4
10.244.5.7      0.0.0.0         255.255.255.255 UH    0      0        0 cali969530ca43d
10.244.5.8      0.0.0.0         255.255.255.255 UH    0      0        0 calib3c61c3cba9
Improve this question

2 Answers 2

Reset to default
0

Do you have vxlan in canal.yaml? Then change that to host-gw and see if it fixes the problem. flannel will communicate new routes to the kernel with netlink sockets then.

  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan" # CHANGE TO host-gw
      }
    }

More info here.

kubernetes access nodeport fails, master cannot reach workers

and on the flannel github.

https://github.com/flannel-io/flannel

If it still doesn't work, try adding the specific network interface flannel must work on. If you dont it will claim the first interface it finds.

 - name: kube-flannel
          image: quay.io/coreos/flannel:v0.9.1
          command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr" ]

add --iface=iface_you_want to "command"

Improve this answer
0

So after deleting all canal PODs and letting them recreate, everything started to work again as it should. Seems like the recreation of those PODs somehow recreated routes and added the Worker node. Not sure why it didnt work right after a worker reboot, later when removing and re-joining this worker to the cluster, it got assigned a new CIDR, after canal POD recreation it started to work.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.