Preamble : i'm not a K8s expert at all as i'm actually learning K8s by doing projects so please be gentle.
Context:
I got a K8s cluster where I setup AWX using the official Helm install. On this cluster, there I also setup a Sonatype Nexus repository with a docker(hosted) private registry configured in order to pull custom AWX EE image I built.
At the moment, the private registry works only with a HTTP listener configured on port 8080.
I'm able to login and push/pull my custom image to this registry from another server using docker cli.
Configuration/AWX Setup for image pulling:
In order to pull my custom image into my AWX project, I configure AWX the following way :
Setup a custom EE
Create a Container Registry credentials
And finally setup Execution Environment on my templates.
Problem:
When i launch a template, it keeps running for a long time, and here's error i got from kubectl :
PS > kubectl describe pods automation-job-50-78mp5
Name: automation-job-50-78mp5
Namespace: awx
Priority: 0
Service Account: default
Node: XXXXXX
Start Time: Tue, 22 Aug 2023 11:42:04 +0200
Labels: ansible-awx=4b53c506-cd99-41df-a4d5-1ed0395e8906
ansible-awx-job-id=50
Annotations: cni.projectcalico.org/containerID: 66b9d3f6852b82b5e1d3b37ad8fc199667170c207f5f79af2c2e5d3a67007d9e
cni.projectcalico.org/podIP: 172.20.5.208/32
cni.projectcalico.org/podIPs: 172.20.5.208/32
Status: Pending
IP: 172.20.5.208
IPs:
IP: 172.20.5.208
Containers:
worker:
Container ID:
Image: awx-repo.mydomain.com/awx-repo
Image ID:
Port: <none>
Host Port: <none>
Args:
ansible-runner
worker
--private-data-dir=/runner
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Requests:
cpu: 250m
memory: 100Mi
Environment: <none>
Mounts: <none>
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes: <none>
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 20s default-scheduler Successfully assigned awx/automation-job-50-78mp5 to frhd01cor003-ebeb10-worker-3
Normal BackOff 19s (x2 over 20s) kubelet Back-off pulling image "awx-repo.mydomain.com/awx-repo"
Warning Failed 19s (x2 over 20s) kubelet Error: ImagePullBackOff
Normal Pulling 5s (x2 over 20s) kubelet Pulling image "awx-repo.mydomain.com/awx-repo"
Warning Failed 5s (x2 over 20s) kubelet Failed to pull image "awx-repo.mydomain.com/awx-repo": rpc error: code = Unknown desc = failed to pull and unpack image "awx-repo.mydomain.com/awx-repo:latest": failed to resolve reference "awx-repo.mydomain.com/awx-repo:latest": failed to do request: Head "https://awx-repo.mydomain.com/v2/awx-repo/manifests/latest": x509: certificate signed by unknown authority
Warning Failed 5s (x2 over 20s) kubelet Error: ErrImagePull
My question are :
- How can I double check the pod is using credentials I set from AWX GUI ?
- As credentials works from the server, is there a "dedicated format" we have to set from AWX GUI ?
- Does the certificate x509 warning may impact the pull action ? (even if my repo seems well configured with a wildcard certificate"
Thanks a lot for your help !
Gael