Debug Docker DNS

Question

In a stack with higher number of services it seems that Docker DNS does not work anymore which leads to effects that container's names are not known even to themselves.

While I have filed this bug, I wonder whether there is a way to debug this more extensively to "prove" the DNS error, or for example manually "fix" the DNS name.

Will be there some deeper error message?

Note: The ping from outside does not work; I just find cumbersome that is not possible to ping your own DNS name.

Answer 1

There is a comprehensive document about k8s' DNS. According to this document one could validate whether the DNS is working by running:

busybox.yaml

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - image: busybox
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    name: busybox
  restartPolicy: Always

and deploy it by issuing:

kubectl create -f busybox.yaml

Once deployed, one could run:

kubectl get pods busybox

and validate whether the DNS is working:

kubectl exec -ti busybox -- nslookup kubernetes.default

There are additional validation steps that could be executed, including:

kubectl exec busybox cat /etc/resolv.conf

verify the DNS policy, check whether the DNS pod runs, checking erros in the DNS pod:

kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c kubedns
kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c dnsmasq
kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c sidecar

does the DNS service run?

kubectl get svc --namespace=kube-system

exposed DNS endpoints?

kubectl get ep kube-dns --namespace=kube-system

There are also multiple known issues regarding the k8s' DNS:

Linux’s libc is impossibly stuck (see this bug from 2005) with limits of just 3 DNS nameserver records and 6 DNS search records. Kubernetes needs to consume 1 nameserver record and 3 search records. This means that if a local installation already uses 3 nameservers or uses more than 3 searches, some of those settings will be lost. As a partial workaround, the node can run dnsmasq which will provide more nameserver entries, but not more search entries. You can also use kubelet’s --resolv-conf flag.