-1

This question seeks a solution for the following errors

If you create an EKS cluster using terraform, and then use the same credentials you used to create the cluster, but see these errors, then what can you do to fix the issue?

AWS Console

Your current IAM principal doesn’t have access to Kubernetes objects on this cluster.
This may be due to the current user or role not having Kubernetes RBAC permissions to describe cluster resources or not having an entry in the cluster’s auth config map

Which includes a not-so-helpful link to Learn more

kubectl

the server has asked for the client to provide credentials

eksctl

Error: getting auth <object>>: Unauthorized
Improve this question

3 Answers 3

Reset to default
1

What is the use case for creating an EKS cluster with enable_cluster_creator_admin_permissions = false?

This is the default when creating a cluster and the recommended way to create it.

If set to true the principal that created the cluster has full admin access. This will probably violate your RBAC concept. Also since the cluster is usually created by a pipeline, this gives the pipeline's principal access to all cluster resources, which it will most probably not need and only increases the attack vector.

Best practice is to have a cluster admin principal be created with the cluster and added as cluster admin on creation.

[...]
enable_cluster_creator_admin_permissions = false
access_entries = {
    cluster_admin = {
      kubernetes_groups = []
      principal_arn     = var.arn

      policy_associations = {
        cluster_admin = {
          policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"
          access_scope = {
            type = "cluster"
          }
        }
      }
    }
    dev_team_one = {
      [...]
    }
}
[...]

By that it becomes far easier to manage access control to your cluster resources.

You can grant the creator full admin access in PoC clusters. But other than that it should be disabled for various reasons.

EDIT

But why no kubernetes groups... shouldn't you add it to system.masters?

kubernetes_groups is meant for custom groups. If you want to bind to system:masters rather use policy_associations as in the example above.

policy_associations.x.access_scope.policy_arn is mapped to one of the user-facing roles

The example above would map to 'cluster-admin' to the system:masters group via ClusterRoleBinding.

You could probably also set the group there and don't set policy_associations. But I have not tested it. Point is, it does the same and is the recommended way.

Improve this answer
4
  • Thank you! But why no kubernetes groups... shouldn't you add it to system.masters? My problem was that with enable_cluster_creator_admin_permissions = false, then even with an IAM principal with full admin access I was a t a dead end because I could not add any IAM principal to system.masters
    – Seth E
    Commented Sep 13 at 12:33
  • I expanded the answer. Please consider accepting it, if it was helpful.
    – Chris
    Commented Sep 13 at 13:19
  • 1
    very cool. And here is where the terraform-aws-eks module executes that very same logic if enable_cluster_creator_admin_permissions = true - github.com/terraform-aws-modules/terraform-aws-eks/blob/master/…
    – Seth E
    Commented Sep 14 at 16:48
  • can you please check with the IAM access for this cluster?
    – MD Nasirul Islam
    Commented Sep 21 at 16:28
1

The Answer

Prerequisite

Ensure the cluster API endpoint is accessible.

If kubectl returns an error like this:

"https://XXXXX.gr7.us-east-1.eks.amazonaws.com/api?timeout=32s": dial tcp 10.0.2.234:443: i/o timeout

...then one possibility preventing you from connecting to the API is API server endpoint access. You can either set it public with a CIDR access list, or must access it from inside the VPC.

see cluster_endpoint_public_access and associated settings here.

Solution

Ensure the cluster creator has admin permissions.

This may be your issue if kubectl returns an error like this:

the server has asked for the client to provide credentials

This one is the big "gotcha", since when creating a cluster with eksctl it takes care of this for you. However, when creating an EKS cluster with Terraform the default for this setting is false. You need to actively configure it to true.

enable_cluster_creator_admin_permissions = true - more info here

Why did I create this post?

I created this question and answer because when I encountered this problem and researched it all of the answers seemed to assume that the cluster was created through eksctl and that the cluster creator already had admin privileges.

The answers I found all assumed that you could actually still issue kubectl and eksctl commands. In the case where you did not specify enable_cluster_creator_admin_permissions = true, kubectl will not work and many eksctl commands will not work and will fail with an error such as this:

Error: getting auth ConfigMap: Unauthorized

Note that some eksctl commands will work, as they rely on the Amazon APIs and not the cluster API. For example these will still work

aws eks --region <region>> update-kubeconfig --name <cluster_name>

eksctl get cluster --name <cluster_name>

The part I do not yet understand

If you create an EKS cluster with the enable_cluster_creator_admin_permissions of false it appears to me that there is no way forward to reconfiguring your cluster to make it usable.

  • You lack the privileges to add a user to system:masters through either the AWS console or eksctl create iamidentitymapping.
  • You cannot use kubectl and cannot access any resources (Cluster, workload, config, service, etc) through the AWS console.

So my remaining question are:

  • What is the use case for creating an EKS cluster with enable_cluster_creator_admin_permissions = false?
  • And if there are none, why is there even a enable_cluster_creator_admin_permissions setting, and why doesn't it default to true?
Improve this answer
1

I had this issue after I launched an EKS cluster with Terraform

In order to hack your way into the cluster you will need to:

  1. Use this command to find the user/role used to create the cluster
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=CreateCluster --region <region>
  1. Update the above role's trust relationship document and add your Principal like so:
"arn:aws:iam::0123456789123:user/terraform"
  1. Use aws cli to assume the above user/role with code like this:
aws sts assume-role \
    --role-arn "arn:aws:iam::0123456789123:role/devops" \
    --role-session-name "eks-access-session"
  1. Use the output key secret-key and token like so:
export AWS_ACCESS_KEY_ID=ASxxxxxxxxxx
export AWS_SECRET_ACCESS_KEY="Rsdfdfsfsfsfsfsfsfsdfsf"
export AWS_SESSION_TOKEN="IsafadfafsdafasfasfafsafafVeryLongTokennnnn"
aws eks --region <region> update-kubeconfig --name my-cluster
  1. If done correctly, now you should be able to run:
kubectl get nodes
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.