Kubernetes Failing with Self Signed Docker Registry Certificate

Question

We have a private Docker registry and we're attempting to get a our Kubernetes cluster to pull images from it, but it's failing with the following error:

  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  104s               default-scheduler  Successfully assigned namespace/service-8599c54df8-592wm to fat2
  Normal   Pulling    67s                kubelet            Pulling image "192.168.100.178:5000/service:ds-1.2"
  Warning  Failed     67s                kubelet            Error: ErrImagePull
  Warning  Failed     67s                kubelet            Failed to pull image "192.168.100.178:5000/service:ds-1.2": rpc error: code = Unknown desc = failed to pull and unpack image "192.168.100.178:5000/service:ds-1.2": failed to resolve reference "192.168.100.178:5000/service:ds-1.2": failed to do request: Head "https://192.168.100.178:5000/v2/service/manifests/ds-1.2": x509: certificate signed by unknown authority (possibly because of "x509: invalid signature: parent certificate cannot sign this kind of certificate" while trying to verify candidate authority certificate "192.168.100.174: hostname")
  Warning  Failed     65s (x2 over 66s)  kubelet            Error: ImagePullBackOff
  Normal   BackOff    65s (x2 over 66s)  kubelet            Back-off pulling image "192.168.100.178:5000/service:ds-1.2"

I have copied the certificates from the Docker registry to /etc/docker/certs.d/192.168.100.178:5000/ and I have copied the secret verbatim from another cluster that has absolutely no problem pulling images from our Docker registry. Regardless, here it is (as a Helm template):

apiVersion: v1
kind: Secret
metadata:
  name: {{ .Values.global.image_pull_secret }}
  namespace: {{ .Values.global.namespace }}
  labels:
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/release-name: {{ .Release.Name }}
    app.kubernetes.io/release-namespace: {{ .Release.Namespace }}
data:
  .dockerconfigjson: {snipped}
type: kubernetes.io/dockerconfigjson

I have verified I can successfully connect with a docker login:

$ docker login https://192.168.100.178:5000
Username: myuser
Password:
WARNING! Your password will be stored unencrypted in /home/myuser/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

I have completely run out of ideas here, but it seems the new cluster doesn't trust the CA that signed the certificate from the Docker registry. I'm at my wits end, so any help here is greatly appreciated.

Answer 1

Finally got it working. I had to install the root certificate by copying it to /etc/ssl/tls/certs. This is the root certificate that was used when creating the certificate for the Docker registry. In the error message, this is the certificate for the CA, which is referenced at the end: "192.168.100.174: hostname".