kubectl uses wrong IP

Question

My kubectl suddenly stopped working. Any commands results in "no route to host".

For example, getting logs kubectl logs mypod-746cfd667b-tdrnt

Error from server: Get "https://hades:10250/containerLogs/default/mypod-746cfd667b-tdrnt/mypod": dial tcp 10.0.0.10:10250: connect: no route to host

That's no wonder, because my server is located at 10.0.0.102, and for some reason, it is going to 10.0.0.10. Why?

• I tried removing kubectl (with purge) and reinstalling it. No difference.
• I tried deleting the ~/.kube folder. No difference.
• I CAN manage my cluster using Kubernetes Lens, which uses the same ~/.kube/config file. However, when trying to get the logs it fallbacks to using kubectl and I get the same issue.

# ping hades

PING hades (10.0.0.102) 56(84) bytes of data.
64 bytes from hades (10.0.0.102): icmp_seq=1 ttl=64 time=0.199 ms

Any idea how I can force kubectl to refresh its DNS cache?

Any idea how I can force kubectl to use a specific IP address, bypassing the DNS entirely?

My ~/.kube/config file is below. Note that "hades" isn't even mentioned anywhere.

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <<<long secret string>>>
    server: https://10.0.0.102:16443
  name: microk8s-cluster
contexts:
- context:
    cluster: microk8s-cluster
    user: admin
  name: microk8s
current-context: microk8s
kind: Config
preferences: {}
users:
- name: admin
  user:
    token: <<<short-secret-string>>>

Answer 1

The issue isn't on this machine at all. When making calls such as 'logs', kubectl contacts the node which in turns looks up which ip it should talk to. In this case, the node has a faulty /etc/hosts file with the last digit 2 dropped:

hades 10.0.0.10

All that is needed is to ssh into the machine and update the hosts file there. There never was an issue on the client machine.