I have an instance of MicroK8s, I am currently migrating my applications. But I can't authenticate to Gitlab's private registry.

About my environment:

  • MicroK8s
  • Gitlab (with registry working securely)

Tests I performed

To test my private Gitlab registry I used another machine and did a docker pull registry.mydomain.com/my-group/my-image:my-tag.

In K8s I followed the instructions available on the K8s website and also the instructions in the answer to this question on SO.

I've already checked the namespaces, the secret was created correctly! I restored the credentials to confirm and they are correct.

Different ways I used to create the secret for imagePullSecrets.

First try

kubectl create secret docker-registry regcred \
  --namespace=my-namespace \
  --docker-server=registry.mydomain.com \
  --docker-username="my-user" \
  --docker-email="my-mail" \

Second attempt

kubectl create secret generic regcred \
 --namespace=my-namespace \
 --from-file=.dockerconfigjson=/home/hsouza/.docker/config.json \

Both ways of creating my secret work, however when creating my pod the ImagePullBackOff error is presented.

For both attempts to generate the regcred I created the POD like this:

apiVersion: v1
kind: Pod
  namespace: my-namespace
  name: test-private-register-pod
  - name: test-private-register-pod
    image: registry.mydomain.com/my-group/my-image:1.0.0
  - name: regcred

When analyzing with kubectl describe pod/test-private-register-pod the following message is returned:

Name:             test-private-register-pod
Namespace:        my-namespace
Priority:         0
Service Account:  default
Node:             k8s-dev/
Start Time:       Sat, 07 Oct 2023 11:25:23 -0300
Labels:           <none>
Annotations:      cni.projectcalico.org/containerID: 0d1a8c2d3f54ea9dfeccb69b96f930a6c5f40d6c9fa8c16994ac24676cecb5be
Status:           Pending
    Container ID:   
    Image:          registry.mydomain.com/my-group/my-image:1.0.0
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:    <none>
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8gj47 (ro)
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  16s                default-scheduler  Successfully assigned my-namespace/test-private-register-pod to k8s-dev
  Normal   Pulling    15s                kubelet            Pulling image "registry.mydomain.com/my-group/my-image:1.0.0"
  Warning  Failed     15s                kubelet            Failed to pull image "registry.mydomain.com/my-group/my-image:1.0.0": rpc error: code = Unknown desc = failed to pull and unpack image "registry.mydomain.com/my-group/my-image:1.0.0": failed to resolve reference "registry.mydomain.com/my-group/my-image:1.0.0": failed to authorize: failed to fetch oauth token: unexpected status: 403 Forbidden
  Warning  Failed     15s                kubelet            Error: ErrImagePull
  Normal   BackOff    14s (x2 over 15s)  kubelet            Back-off pulling image "registry.mydomain.com/my-group/my-image:1.0.0"
  Warning  Failed     14s (x2 over 15s)  kubelet            Error: ImagePullBackOff

In the gitlab nginx logs, the status code for K8s access is 401. I put it in a fictitious way to try to pull from https://requestcatcher.com/ and when analyzing the request made by K8s it does not send the header with Basic Auth .

This is my problem, I appreciate the community's help in trying to identify where I'm going wrong.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.