AWX Failed to pull image on a private registry

Question

Context:
I got a K8s cluster where I setup AWX using the official Helm install. On this cluster, there I also setup a Sonatype Nexus repository with a docker(hosted) private registry configured in order to pull custom AWX EE image I built.

At the moment, the private registry works only with a HTTP listener configured on port 8080. I'm able to login and push/pull my custom image to this registry from another server using docker cli.

Configuration/AWX Setup for image pulling:
In order to pull my custom image into my AWX project, I configure AWX the following way :

1. Setup a custom EE

2. Create a Container Registry credentials
   

3. And finally setup Execution Environment on my templates.

Problem:

When i launch a template, it keeps running for a long time, and here's error i got from kubectl :

PS > kubectl describe pods automation-job-50-78mp5
Name:             automation-job-50-78mp5
Namespace:        awx
Priority:         0
Service Account:  default
Node:             XXXXXX
Start Time:       Tue, 22 Aug 2023 11:42:04 +0200
Labels:           ansible-awx=4b53c506-cd99-41df-a4d5-1ed0395e8906
                  ansible-awx-job-id=50
Annotations:      cni.projectcalico.org/containerID: 66b9d3f6852b82b5e1d3b37ad8fc199667170c207f5f79af2c2e5d3a67007d9e
                  cni.projectcalico.org/podIP: 172.20.5.208/32
                  cni.projectcalico.org/podIPs: 172.20.5.208/32
Status:           Pending
IP:               172.20.5.208
IPs:
  IP:  172.20.5.208
Containers:
  worker:
    Container ID:
    Image:         awx-repo.mydomain.com/awx-repo
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Args:
      ansible-runner
      worker
      --private-data-dir=/runner
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:        250m
      memory:     100Mi
    Environment:  <none>
    Mounts:       <none>
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:            <none>
QoS Class:          Burstable
Node-Selectors:     <none>
Tolerations:        node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                    node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  20s                default-scheduler  Successfully assigned awx/automation-job-50-78mp5 to frhd01cor003-ebeb10-worker-3
  Normal   BackOff    19s (x2 over 20s)  kubelet            Back-off pulling image "awx-repo.mydomain.com/awx-repo"
  Warning  Failed     19s (x2 over 20s)  kubelet            Error: ImagePullBackOff
  Normal   Pulling    5s (x2 over 20s)   kubelet            Pulling image "awx-repo.mydomain.com/awx-repo"
  Warning  Failed     5s (x2 over 20s)   kubelet            Failed to pull image "awx-repo.mydomain.com/awx-repo": rpc error: code = Unknown desc = failed to pull and unpack image "awx-repo.mydomain.com/awx-repo:latest": failed to resolve reference "awx-repo.mydomain.com/awx-repo:latest": failed to do request: Head "https://awx-repo.mydomain.com/v2/awx-repo/manifests/latest": x509: certificate signed by unknown authority
  Warning  Failed     5s (x2 over 20s)   kubelet            Error: ErrImagePull

My question are :

• How can I double check the pod is using credentials I set from AWX GUI ?
• As credentials works from the server, is there a "dedicated format" we have to set from AWX GUI ?
• Does the certificate x509 warning may impact the pull action ? (even if my repo seems well configured with a wildcard certificate"

Answer 1

After days and tests here's the solution :

Kubernetes worker and master got the wrong certificate.

In my case, i have to :

• create a .pem certificate chain composed of the intermediate + wildcard certificate
• convert the .pem chain into a .pem fileas my tool (Nutanix Kubernetes Engine) need a .pem format.

Finally, workers and master got certificate updated into /etc/containerd/certs.d/my_repo_folder.

This makes AWX working and successfully pull image from my private registry.