No Such Host: Kubernetes/Docker cannot pull from private k8 registry

Question

I have a private docker registry in k8 in the default namespace with tls at https://docker-registry.default:5000. By doing local port forwarding to it(at port 5000) and adding docker-registry.default to my /etc/hosts file, I have been able to pull and push images to it.

However, I'm using a serverless framework(Nuclio) which manages the deploy for me. The error I get is:

Normal   Scheduled  7m52s                   default-scheduler                                       Successfully assigned nuclio/helloworld1-96876fcf9-7c87w to gke-your-first-cluster-1-pool-1-fe915942-gnkx
  Normal   Pulling    6m14s (x4 over 7m51s)   kubelet, gke-your-first-cluster-1-pool-1-fe915942-gnkx  Pulling image "docker-registry.default:5000/docker/nuclio/processor-helloworld1:latest"
  Warning  Failed     6m14s (x4 over 7m51s)   kubelet, gke-your-first-cluster-1-pool-1-fe915942-gnkx  Failed to pull image "docker-registry.default:5000/docker/nuclio/processor-helloworld1:latest": rpc error: code = Unknown desc = Error response from daemon: Get https://docker-registry.default:5000/v2/: dial tcp: lookup docker-registry.default on 169.254.169.254:53: no such host
  Warning  Failed     6m14s (x4 over 7m51s)   kubelet, gke-your-first-cluster-1-pool-1-fe915942-gnkx  Error: ErrImagePull
  Normal   BackOff    6m2s (x6 over 7m51s)    kubelet, gke-your-first-cluster-1-pool-1-fe915942-gnkx  Back-off pulling image "docker-registry.default:5000/docker/nuclio/processor-helloworld1:latest"
  Warning  Failed     2m38s (x21 over 7m51s)  kubelet, gke-your-first-cluster-1-pool-1-fe915942-gnkx  Error: ImagePullBackOff

It gives a no such host error.

A Nuclio developer has confirmed that it's not a nuclio issue since I cannot even do a docker login from the Nuclio Admin pod(not the failed pod above) to this internal docker-registry.default though I can do a docker login to the Docker hub and a wget to docker-registry.default successfully. The Nuclio Admin pod is Alpine Linux if it helps.

Answer 1

It seems your private DNS is misconfigured. Check your host or make sure that docker-registry.default is the a valid hostname.

You can run a temporary Ubuntu pod to check if that hostname is resolving correctly from your cluster, e.g.

$ kubectl run ubuntu -it --image ubuntu --rm=true -- bash
root@shell:/# apt update && DEBIAN_FRONTEND=noninteractive apt --yes install dnsutils
root@shell:/# dig docker-registry.default
docker-registry.default.    IN  A D0.y0u.h4ve.1P?

Additional suggestions:

• If you're using a cloud provider (such as Azure), make sure your Private DNS zone is linked with your cluster's virtual network. Read: Connect privately to an Azure container registry using Azure Private Link
• If your container registry endpoint isn't correct, edit the deployment manifest and fix the image path.
• Test pulling the image from another VM within the same private network.