Amazon EKS from local kubectl does not cache MFA token/session

Question

I have set up an EKS cluster in AWS. My user-account there is secured with MFA. And I have an assumerole, that is allowed to manage my infrastructure.

So I installed and configured my system as stated in the AWS userguide: https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html

I have kubectl in version 1.11.x installed, so I can use that binary to communicate with aws (and I do not need the patched amazon version). I then installed aws-cli and aws-iam-authenticator.

user@notebook [04:44:51 PM] [~]
-> % kubectl version --short --client
Client Version: v1.11.0
user@notebook [04:45:13 PM] [~]
-> % aws-iam-authenticator help
A tool to authenticate to Kubernetes using AWS IAM credentials
...
user@notebook [04:45:25 PM] [~]
-> % aws --version
aws-cli/1.15.51 Python/3.6.6 Linux/4.17.8-1-ARCH botocore/1.10.50
user@notebook [04:45:30 PM] [~]
-> %

I then created the configuration files from the how-to ~/.aws/credentials, ~/.aws/config, ~/.kube_aws/config and created an alias for myself to use kubectl with the new config (since I have more clusters to manage): alias kubectlaws='KUBECONFIG=/home/user/.kube_aws/config kubectl $@'.

So here comes my problem: When I connect to the cluster using the aws-cli binary, I have to enter my MFA token and then the session is cached (I read for about 15 minutes).

user@notebook [04:58:52 PM] [~]
-> % aws eks describe-cluster --profile my_profile --name ClusterName  --query cluster.status                                                                                          
Enter MFA code for arn:aws:iam::<MY_MFA_ARN>:
"ACTIVE"
user@notebook [04:59:09 PM] [~]
-> % aws eks describe-cluster --profile my_profile --name ClusterName  --query cluster.endpoint                                                                                        
"https://..."
user@notebook [04:59:30 PM] [~]
-> %

But when I do this with my kubectl alias, I have to enter the token every time I run a command! Furthermore, I cannot re-use a code, so I have to wait about 30 seconds, until the next one is provided.

user@notebook [05:03:56 PM] [~]
-> % kubectlaws get nodes
Assume Role MFA token code: 123456
NAME                             STATUS    ROLES     AGE       VERSION
...
user@notebook [05:04:05 PM] [~]
-> % kubectlaws get nodes
Assume Role MFA token code: 123456
could not get token: AccessDenied: MultiFactorAuthentication failed with invalid MFA one time pass code.                                                                                      
        status code: 403, request id: <SOME_ID>
No resources found.
Unable to connect to the server: getting credentials: exec: exit status 1
user@notebook [05:04:14 PM] [~]
-> %

This is really annoying, as you might imagine. Has anyone any idea what I did wrong there? Is this a bug? Or is this some kind of mis-configuration?

Thanks in advance!

Answer 1

Indeed, this is a pain. I've been fighting with it for the whole day yesterday, but managed to get it working, let's see if our cases are similar enough.

My guess is that the usual aws configure configuration with a MFA ARN + the AWS SDK used to develop kubectl are not entirely compatible: I have two profiles set for this, one that has its mfa_serial_arn and role_arn that works as follows:

• First aws command I type, I get prompted for MFA token
• From there onwards, while session is active, I can work without typing in more tokens.
• kubectl asks for a token EVERY SINGLE TIME.

This is bad.

Now, I set a different profile, without any of the above, and I do the following:

```

keys=($(aws sts assume-role --role-arn <ARN_OF_DELEGATED_ROLE> \
   --role-session-name  <RANDOM_UNIQUE_NAME> \
   --serial-number <ARN_OF_MFA_DEVICE>\
   --token-code  \
    --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' --output text))

# profile used here will be configured in ~/.aws/credentials
export AWS_PROFILE=<PROFILE YOU WANT>

aws configure set aws_access_key_id ${temp_keys[0]}
aws configure set aws_secret_access_key ${temp_keys[1]}
aws configure set aws_session_token ${temp_keys[2]}

```

This creates a temporary valid aws profile, that can be used BOTH with the aws cli, and kubectl.

Answer 2

Btw: I managed to fix this problem by using aws-mfa (https://github.com/broamski/aws-mfa), which I have configured to cache my aws credentials for an hour.

Answer 3

For anyone came here to solve this problem without any workarounds - the solution is simple.

We need to add switch --cache to the kubeconfig for a user used to access a Kubernetes cluster.

users:
- name: <redacted>
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - token
      - --cache
      - -i
      - prod

Tested on:

$ aws-iam-authenticator version
{"Version":"v0.5.1","Commit":"d7c0b2e9131faabb2b09dd804a35ee03822f8447"}

Hope this will help someone.