2

I setup a Kubernetes cluster using Kubespray. it works flawlessly until recently I encountered something strange.

I created a BIND DNS deployment using the following manifest.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bind
  labels:
    app: bind
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bind
  template:
    metadata:
      labels:
        app: bind
    spec:
      containers:
      - name: bind
        image: sameersbn/bind:9.16.1-20200524
        ports:
        - containerPort: 53
          name: tcp
          protocol: TCP 
        - containerPort: 53
          name: udp
          protocol: UDP
        - containerPort: 10000
          name: web
          protocol: TCP
        volumeMounts:
        - name: bind-data
          mountPath: /data
      volumes:
      - name: bind-data
        hostPath:
          path: /data
---
apiVersion: v1
kind: Service
metadata:
  name: bind
spec:
  type: ClusterIP
  ports:
  - port: 53
    name: tcp
    targetPort: tcp
    protocol: TCP
  - port: 53
    name: udp
    targetPort: udp
    protocol: UDP
  - port: 10000
    name: web
    targetPort: web
    protocol: TCP
  selector:
    app: bind

Then I've defined a master zone and created a forward dns record in it.

test.arman.local -> 192.168.230.216

Next I have configured the configmap of coredns to use it:

kubectl edit configmap -n kube-system coredns

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
            lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    arman.local:53 {
        errors
        cache 30
        forward . 10.233.28.178
    }
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"Corefile":".:53 {\n    errors {\n    }\n    health {\n        lameduck 5s\n    }\n    ready\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n      pods insecure\n      fallthrough in-addr.arpa ip6.arpa\n    }\n    prometheus :9153\n    forward . /etc/resolv.conf {\n      prefer_udp\n      max_concurrent 1000\n    }\n    cache 30\n\n    loop\n    reload\n    loadbalance\n}\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"EnsureExists"},"name":"coredns","namespace":"kube-system"}}
  creationTimestamp: "2024-08-08T10:26:32Z"
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
  name: coredns
  namespace: kube-system
  resourceVersion: "2135167"
  uid: 5457c863-3a34-4490-badb-036f4f0e1739

The service of BIND deployment is like:

kubectl get svc                 
NAME                          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
bind                          ClusterIP   10.233.28.178   <none>        53/TCP,53/UDP,10000/TCP      44h

The problem is it can't resolve test.arman.local

I've started to debug the problem I launched a busybox and used nslookup

kubectl run -it busybox --image=busybox:1.28 --restart=Never -- sh

Here is the content of /etc/resolv.conf of busybox

/ # cat /etc/resolv.conf 
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 169.254.25.10
options ndots:5

I don't know what IP 169.254.25.10 is for because the service of dnsCore has different IP:

kubectl get svc -n kube-system 
NAME      TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
coredns   ClusterIP   10.233.0.3   <none>        53/UDP,53/TCP,9153/TCP   10d

In the busybox I can successfully resolve the test.arman.local using the following command:

nslookup test.arman.local 10.233.28.178
Server:    10.233.28.178
Address 1: 10.233.28.178

Name:      test.arman.local
Address 1: 192.168.230.216
/ # ^C
/ # cat /etc/resolv.conf 
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 169.254.25.10
options ndots:5

Or even use this:

nslookup test.arman.local 10.233.0.3 # This is for coredns
Server:    10.233.0.3
Address 1: 10.233.0.3 coredns.kube-system.svc.cluster.local

Name:      test.arman.local
Address 1: 192.168.230.216

We are using DNS/calico, for the sake of completeness I post the pods in kube-system:

 kubectl get pods -n kube-system 
NAME                                      READY   STATUS    RESTARTS        AGE
calico-kube-controllers-c7cc688f8-rb59q   1/1     Running   1 (7d21h ago)   10d
calico-node-5dglq                         1/1     Running   0               10d
calico-node-c592m                         1/1     Running   0               10d
calico-node-hbfqj                         1/1     Running   0               10d
calico-node-s6jrl                         1/1     Running   0               10d
coredns-857f949bf7-bprlf                  1/1     Running   0               65m
coredns-857f949bf7-r9p87                  1/1     Running   0               65m
dns-autoscaler-6ffb84bd6-fgdtz            1/1     Running   0               10d
kube-apiserver-node1                      1/1     Running   1               10d
kube-apiserver-node2                      1/1     Running   1               10d
kube-controller-manager-node1             1/1     Running   4 (7d21h ago)   10d
kube-controller-manager-node2             1/1     Running   7 (6d19h ago)   10d
kube-proxy-hb2wn                          1/1     Running   0               10d
kube-proxy-j5pfg                          1/1     Running   0               10d
kube-proxy-qjj4n                          1/1     Running   0               10d
kube-proxy-rp9mt                          1/1     Running   0               10d
kube-scheduler-node1                      1/1     Running   4 (6d19h ago)   10d
kube-scheduler-node2                      1/1     Running   6 (6d19h ago)   10d
nginx-proxy-node3                         1/1     Running   0               10d
nginx-proxy-node4                         1/1     Running   0               10d
nodelocaldns-6sztv                        1/1     Running   0               10d
nodelocaldns-bpt7s                        1/1     Running   0               10d
nodelocaldns-n4rsk                        1/1     Running   0               10d
nodelocaldns-n8q9w                        1/1     Running   0               10d

UPDATE: I found out it has something to do with nodelocaldns but I can not resolve the issue

Improve this question
1

2 Answers 2

Reset to default
0

I managed to solve the issue by changing the configuration of nodelocaldns:

kubectl edit configmap -n kube-system nodelocaldns

change from

cluster.local:53 {
    errors
    cache {
        success 9984 30
        denial 9984 5
    }
    reload
    loop
    bind 169.254.25.10
    forward . 10.233.0.3 {
        force_tcp
    }
    prometheus :9253
    health 169.254.25.10:9254
}
in-addr.arpa:53 {
    errors
    cache 30
    reload
    loop
    bind 169.254.25.10
    forward . 10.233.0.3 {
        force_tcp
    }
    prometheus :9253
}
ip6.arpa:53 {
    errors
    cache 30
    reload
    loop
    bind 169.254.25.10
    forward . 10.233.0.3 {
        force_tcp
    }
    prometheus :9253
}
.:53 {
    errors
    cache 30
    reload
    loop
    bind 169.254.25.10
    forward . /etc/resolv.conf
    prometheus :9253
}

to this:

cluster.local:53 {
    errors
    cache {
        success 9984 30
        denial 9984 5
    }
    reload
    loop
    bind 169.254.25.10
    forward . 10.233.0.3 {
        force_tcp
    }
    prometheus :9253
    health 169.254.25.10:9254
}
in-addr.arpa:53 {
    errors
    cache 30
    reload
    loop
    bind 169.254.25.10
    forward . 10.233.0.3 {
        force_tcp
    }
    prometheus :9253
}
ip6.arpa:53 {
    errors
    cache 30
    reload
    loop
    bind 169.254.25.10
    forward . 10.233.0.3 {
        force_tcp
    }
    prometheus :9253
}
.:53 {
    errors
    cache 30
    reload
    loop
    bind 169.254.25.10
    forward . 10.233.0.3 {
        force_tcp
    }
    prometheus :9253
}

look at the last block

I am curious what caused this issue in the first place?

Improve this answer
0

In Kubernetes there is a convention that the cluster DNS service is on the .10 IP address.

So it's expected that you'll configure the ClusterIP for CoreDNS to request that particular IP.

If not, you'll need to configure kubelet on each node to change what it injects into the /etc/resolv.conf file of most pods.

Improve this answer
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.