How to fix wrong /etc/resolv.conf in kubernetes pods after creating a cluster with EKS eksctl with vpc-cni plugin?

Question

So, i've created a kubernetes cluster in EKS with eksctl, ClusterConfig:

apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig

metadata:
  name: exchange-develop
  region: us-east-1
  version: '1.19'

managedNodeGroups:

  - name: custom-bootstrap
    labels:  
      worker: default
    instanceType: m5.large
    desiredCapacity: 2
    minSize: 2
    maxSize: 2
    ami: ami-00836a7940260f6dd
    tags:
      'k8s.io/cluster-autoscaler/enabled': 'true'
      'k8s.io/cluster-autoscaler/exchange-develop': 'owned'
    iam: 
     attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::658464581062:policy/eks-csi-driver-policy
        - arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess
        - arn:aws:iam::658464581062:policy/ALBIngressControllerIAMPolicy
        - arn:aws:iam::658464581062:policy/ExternalDNSPlicy
        - arn:aws:iam::658464581062:policy/eks-cluster-autoscaler
    # maxPodsPerNode: 110
    overrideBootstrapCommand: |
      #!/bin/bash
      /etc/eks/bootstrap.sh exchange-develop --kubelet-extra-args '--node-labels=eks.amazonaws.com/sourceLaunchTemplateVersion=1,alpha.eksctl.io/cluster-name=exchange-develop,alpha.eksctl.io/nodegroup-name=custom-bootstrap,eks.amazonaws.com/nodegroup-image=ami-00836a7940260f6dd,eks.amazonaws.com/capacityType=ON_DEMAND,eks.amazonaws.com/nodegroup=custom-bootstrap,worker=default,eks.amazonaws.com/sourceLaunchTemplateId=lt-0dae52a718d4ba6e5 --max-pods=110' --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL --dns-cluster-ip $K8S_CLUSTER_DNS_IP --use-max-pods false



availabilityZones: ['us-east-1c', 'us-east-1d']

iam:
  withOIDC: true

vpc:
  cidr: 10.10.0.0/16
  clusterEndpoints:
    publicAccess: true
    privateAccess: true

addons:
- name: vpc-cni
  version: '1.10.1'
  attachPolicyARNs:
    - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy

Then I can see the pod IP's are different from the vpc cidr, after recovering everything from another development cluster with Velero, there is no DNS resolving, so when checking the contents of /etc/resolv.conf in any pod the content is:

nameserver 10.10.0.2
search ec2.internal
options timeout:2 attempts:5

When checking dns-service endpoints:

NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
kube-dns         ClusterIP   172.20.0.10      <none>        53/UDP,53/TCP   21h

Endpoints:

Name:         kube-dns
Namespace:    kube-system
Labels:       eks.amazonaws.com/component=kube-dns
              k8s-app=kube-dns
              kubernetes.io/cluster-service=true
              kubernetes.io/name=CoreDNS
Annotations:  endpoints.kubernetes.io/last-change-trigger-time: 2021-12-06T19:38:03Z
Subsets:
  Addresses:          10.10.34.2,10.10.61.97
  NotReadyAddresses:  <none>
  Ports:
    Name     Port  Protocol
    ----     ----  --------
    dns-tcp  53    TCP
    dns      53    UDP

Events:  <none>

Answer 1

Related issue: https://github.com/awslabs/amazon-eks-ami/issues/636

So, not specifying --dns-cluster-ip, --b64-cluster-ca and --apiserver-endpoint fixed the DNS issue.

Final bootstrap command:

  /etc/eks/bootstrap.sh exchange-develop --kubelet-extra-args '--node-labels=eks.amazonaws.com/sourceLaunchTemplateVersion=1,alpha.eksctl.io/cluster-name=exchange-develop,alpha.eksctl.io/nodegroup-name=custom-ami-bootstrap-ssh,eks.amazonaws.com/nodegroup-image=ami-0082fe83ca6d7cf7d,eks.amazonaws.com/capacityType=ON_DEMAND,eks.amazonaws.com/nodegroup=custom-bootstrap,worker=default --max-pods=110' --use-max-pods false