How to resolve a problem "certificate signed by unknown authority" in GKE on pulling image (a private registry) when a pod is created?

Question

I generated a CA certificate, then issued a certificate based on it for a private registry, that located in the same GKE cluster. Put the server certificates to the private registry and the CA certificate to all GKE nodes and run:

update-ca-certificates && systemctl restart docker

Images are building and putting into the private registry without problems. When a pod tries to pull the an image from the repository I get an error:

x509: certificate signed by unknown authority

Also I tried to put the CA certificate to the docker certs.d directory (10.3.240.100:3000 — the IP address of the private registry) and restart the docker on each node of the GKE cluster, but it doesn't help, too:

/etc/docker/certs.d/10.3.240.100:3000/ca.cert

How to solve this problem? Am I understand correctly that the GKE nodes' docker is responsible for pulling images when creating a pod?

Answer 1

If your on Standard Mode for your GKE cluster, go into Computer Engine > SSH into every node and put into /etc/docker/certs.d/<ip:port>/ca.cert. Create folder if it does not exists.

This works for me. The method will not work for Autopilot Mode.

My exact steps on each node:

mkdir -p /etc/docker/certs.d/1.2.3.4:5000  
cd /etc/docker/certs.d/1.2.3.4:5000  
vi /etc/docker/certs.d/1.2.3.4:5000/ca.crt  

Lastly, create and use the imagepullsecrets in your deployment

kubectl create secret docker-registry my-secret \ 
--docker-server=1.2.3.4:5000 \
--docker-username='testuser' \
--docker-password='testpassword' \
--docker-email='[email protected]'

cat <<EOF | kubectl apply -f 
...
 spec:  
   containers:  
   - name: hello  
     image: 1.2.3.4:5000/test:7.12.0  
   imagePullSecrets:  
   - name: my-secret   
EOF

Answer 2

I found a solution. The problem is actual for Kubernetes version 1.19+ and COS/Ubuntu images based on containerd for GKE nodes. Before the 1.19 version Kubernetes used to use Docker for building images, but now it uses containerd. More details could be found in the official Google Cloud documentation.

You need to create and put an CA certificate to each GKE node. Here you can find an answer how to do it correctly https://stackoverflow.com/a/67724696/3319341.

But for containerd solution you should replace command

nsenter --target 1 --mount systemctl restart docker

by

nsenter --target 1 --mount systemctl restart containerd

A more detailed answer: https://stackoverflow.com/a/67990395/3319341

Answer 3

You can now use the DaemonSet to load self-signed CAs when using containerd: https://github.com/samos123/gke-node-ca-importer

Credit to @nstogner: https://github.com/samos123/gke-node-ca-importer/commit/756674ee595e1cf86df4b3181bb3cf687bd72c97