2

In Kubernetes, if I run a Service Foo on every Node, and also Service Bar on every Node, i.e. with DaemonSets. If a Pod from Service Foo needs to make a request to Service Bar, does Kubernetes networking magic optimise things so communication will be done within the same Node, i.e. without a external network call? Or will a random Pod Bar be picked meaning usually Bar will be on a different Node, so there will be a network hop?

Context to question: I'm thinking to run CoreDNS as a Daemonset, because with the cache plugin, I'm hoping most DNS queries will not need to use the network as there will always be a CoreDNS Pod co-located within the same Node and most requests will be cached. Currently we only run 3 CoreDNS pods and we have 12 Nodes and we see some DNS failures at times of network traffic surges - particularly during a deployment when lot of logging is done, which have to be shipped across the network to ElasticSearch.

In case specifics matter:

  • The CoreDNS service runs as clusterIP: 172.20.0.10
  • The services that use CoreDNS run as type: NodePort
  • The Kubernetes cluster is AWS EKS, v1.14.9. Networking is done with the AWS VPC CNI.
Improve this question
1
  • 1
    Have you tried DNS Auto-scaling if that all what you need to get past the dns failure during network surge ..
    – DT.
    Commented Feb 12, 2020 at 9:34

2 Answers 2

Reset to default
3

No, Kubernetes will load-balance requests to a ClusterIP across all Pods with matching labels (and passing readiness checks). It won't keep traffic within the same Node.

Q: So how does the NodeLocal DNS Cache achieve to keep DNS requests local to the Node?

A: When using the NodeLocal DNS Cache, it is necessary to pass a extra arg to the Kubelet of each Node (e.g. --kubelet-extra-args "--cluster-dns=169.254.20.10 ..."). This changes the resolv.conf of every container in every Pod to use DNS resolver on the Node, instead of a clusterIP. The NodeLocal Cache runs on every Node (as a DaemonSet) and uses a arg (e.g. "-localip", "169.254.20.10") to run on the specific IP set in the resolv.conf of all Pods. The NodeLocal DNS Cache Pod also needs to be configured with hostNetwork: true.

Improve this answer
1
  • 1
    Thanks @Tom for crisp and Short explanation.
    – DT.
    Commented Feb 14, 2020 at 8:58
2

This looks to be answered on offcail docs here

NodeLocal DNSCache improves Cluster DNS performance by running a dns caching agent on cluster nodes as a DaemonSet.

With this architecture, Pods will reach out to the dns caching agent running on the same node, thereby avoiding iptables DNAT rules and connection tracking.

Having a local cache will help improve the latency.

[]

Image Reference : https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/#architecture-diagram

Improve this answer
1
  • Thank you. We trialed using the NodeLocal DNSCache but found it complicated our network and we saw a increase in DNS timeouts - we never figured out why. I was wondering whether scaling CoreDNS to every Node would give a similar benefit without the increase in complexity.
    – Tom
    Commented Feb 12, 2020 at 13:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.