2

Using the eksctl tool, I created an EKS cluster with 5 nodes. My application's docker images are stored in ECR registries in the same region. I deployed my kubernetes cluster and everything has been happy for the past 6 weeks or so.

This morning, I came in and found 3 pods were in an ErrImagePull state. Using kubectl describe pod <pod name>, I found the error:

Failed to pull image "<ECR REGISTRY>/<IMAGE>": rpc error: code = Unknown desc = Error response from daemon: Get <ECR REGISTRY>/<IMAGE>: no basic auth credentials

My understanding of EKS and ECR is that I don't need a pull secret (and I haven't used one for any of the other running pods) so my guess is that some process or docker image on that node died but I can't find any docs on this.

Update: I forgot all about this question. When I created the original node group, I failed to include the --ssh-access flag which prevented me from getting onto the node and see if a kubernetes process had failed. I never found the actual solution; I simply added a taint to the problem node, created a new node, and went about my business. I'm still trying to find time to spin up a new node group with ssh access.

2
  • Did you find a solution to this? – Chris F Feb 16 at 1:21
  • Nope. Not related to this probe, we started running an internal docker registry with nexus instead. – Matthew Feb 17 at 23:26
1

Do your IAM roles that are attached to EC2 instances that are in EKS cluster have ECR iam policies? If not please update IAM roles Ref Link: https://docs.aws.amazon.com/AmazonECR/latest/userguide/ECR_on_EKS.html#:~:targetText=The%20Amazon%20EKS%20worker%20node,policy%20permissions%20for%20Amazon%20ECR.&targetText=When%20referencing%20an%20image%20from,tag%20naming%20for%20the%20image.

1
  • Yes, the IAM role has the correct permissions. – Matthew Nov 12 '19 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.