Unable to pull image from a private Docker registry with Let's Encrypt certificate in Kubernetes

Question

I have set up a private Docker registry on my Kubernetes cluster (k3s) running on a Raspberry Pi 4. I have secured the registry with a Let's Encrypt certificate, and I can confirm that the certificate is valid when accessing the registry from a web browser. However, when trying to deploy an application in Kubernetes that uses an image from this private registry, I am encountering an error related to the certificate.

Failed to pull image "192.168.42.50:5000/dev-backend:latest": rpc error: code = Unknown desc = failed to pull and unpack image "192.168.42.50:5000/dev-backend:latest": failed to resolve reference "192.168.42.50:5000/dev-backend:latest": failed to do request: Head "https://192.168.42.50:5000/v2/dev-backend/manifests/latest": x509: certificate signed by unknown authority

I have created a Certificate resource in the "docker" namespace and successfully obtained a certificate from Let's Encrypt. The certificate request and its corresponding certificate can be seen with the following commands:

root@atlas00:~# kubectl get certificaterequests -n docker
NAME                        APPROVED   DENIED   READY   ISSUER             REQUESTOR                                         AGE
docker-registry-tls-hhm6j   True                True    letsencrypt-prod   system:serviceaccount:cert-manager:cert-manager   7h17m

root@atlas00:~# kubectl get certificates -n docker
NAME                  READY   SECRET                AGE
docker-registry-tls   True    docker-registry-tls   7h17m

The registry is using the Let's Encrypt certificate correctly, and I have even added the Let's Encrypt root certificate (ISRG Root X1) to the trusted certificate store for the container runtime (Docker) on each node of my cluster.

Here is my deployment YAML file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dev-backend
  namespace: dev
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dev-backend
  template:
    metadata:
      labels:
        app: dev-backend
    spec:
      containers:
      - name: dev-backend
        image: 192.168.42.50:5000/dev-backend:latest
        ports:
        - containerPort: 80
        volumeMounts:
        - name: dev-backend-storage
          mountPath: /app/data
      volumes:
      - name: dev-backend-storage
        persistentVolumeClaim:
          claimName: dev-backend-pvc
      imagePullSecrets:
      - name: docker-registry-config

I have created a Kubernetes secret (docker-registry-config) containing the necessary credentials for accessing the private registry.

Despite these configurations, Kubernetes still reports the certificate as signed by an unknown authority. How can I ensure that Kubernetes trusts the Let's Encrypt certificate and successfully pulls the image from my private registry? Or is there a certificate authority that Kubernetes trusts out of the box? I'd like to avoid using an 'insecure-registry' approach. Thanks.

Answer 1

I had two problems. @taleodor was correct though, I needed to connect using my domain name that I had setup my certificate for. I put a copy of this question on stackoverflow as well as it seemed there was a bigger body of kubernetes questions there already. I outlined the other problem here https://stackoverflow.com/questions/76146713/unable-to-pull-image-from-a-private-docker-registry-with-lets-encrypt-certifica?noredirect=1#comment134287667_76146713 For this problem I had to find this in my app's deployment yaml:

image: 192.168.42.50:5000/dev-backend:latest

and change it to

image: <my domain>:5000/dev-backend:latest

And apply it of course. Then I had to make sure that port 5000 was port forwarded to my machine at 192.168.42.50 but that got me to a 401 (bad password error) which is basically a success.

You also want to make sure that the domain is resolving properly from the machines running k3s cluster with nslookup <your domain>. If not you probably have a problem with your A record.