3

I have set up an EKS cluster in AWS. My user-account there is secured with MFA. And I have an assumerole, that is allowed to manage my infrastructure.

So I installed and configured my system as stated in the AWS userguide: https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html

I have kubectl in version 1.11.x installed, so I can use that binary to communicate with aws (and I do not need the patched amazon version). I then installed aws-cli and aws-iam-authenticator.

user@notebook [04:44:51 PM] [~]
-> % kubectl version --short --client
Client Version: v1.11.0
user@notebook [04:45:13 PM] [~]
-> % aws-iam-authenticator help
A tool to authenticate to Kubernetes using AWS IAM credentials
...
user@notebook [04:45:25 PM] [~]
-> % aws --version
aws-cli/1.15.51 Python/3.6.6 Linux/4.17.8-1-ARCH botocore/1.10.50
user@notebook [04:45:30 PM] [~]
-> %

I then created the configuration files from the how-to ~/.aws/credentials, ~/.aws/config, ~/.kube_aws/config and created an alias for myself to use kubectl with the new config (since I have more clusters to manage): alias kubectlaws='KUBECONFIG=/home/user/.kube_aws/config kubectl $@'.

So here comes my problem: When I connect to the cluster using the aws-cli binary, I have to enter my MFA token and then the session is cached (I read for about 15 minutes).

user@notebook [04:58:52 PM] [~]
-> % aws eks describe-cluster --profile my_profile --name ClusterName  --query cluster.status                                                                                          
Enter MFA code for arn:aws:iam::<MY_MFA_ARN>:
"ACTIVE"
user@notebook [04:59:09 PM] [~]
-> % aws eks describe-cluster --profile my_profile --name ClusterName  --query cluster.endpoint                                                                                        
"https://..."
user@notebook [04:59:30 PM] [~]
-> %

But when I do this with my kubectl alias, I have to enter the token every time I run a command! Furthermore, I cannot re-use a code, so I have to wait about 30 seconds, until the next one is provided.

user@notebook [05:03:56 PM] [~]
-> % kubectlaws get nodes
Assume Role MFA token code: 123456
NAME                             STATUS    ROLES     AGE       VERSION
...
user@notebook [05:04:05 PM] [~]
-> % kubectlaws get nodes
Assume Role MFA token code: 123456
could not get token: AccessDenied: MultiFactorAuthentication failed with invalid MFA one time pass code.                                                                                      
        status code: 403, request id: <SOME_ID>
No resources found.
Unable to connect to the server: getting credentials: exec: exit status 1
user@notebook [05:04:14 PM] [~]
-> %

This is really annoying, as you might imagine. Has anyone any idea what I did wrong there? Is this a bug? Or is this some kind of mis-configuration?

Thanks in advance!

1

Indeed, this is a pain. I've been fighting with it for the whole day yesterday, but managed to get it working, let's see if our cases are similar enough.

My guess is that the usual aws configure configuration with a MFA ARN + the AWS SDK used to develop kubectl are not entirely compatible: I have two profiles set for this, one that has its mfa_serial_arn and role_arn that works as follows:

  • First aws command I type, I get prompted for MFA token
  • From there onwards, while session is active, I can work without typing in more tokens.
  • kubectl asks for a token EVERY SINGLE TIME.

This is bad.

Now, I set a different profile, without any of the above, and I do the following:

```

keys=($(aws sts assume-role --role-arn <ARN_OF_DELEGATED_ROLE> \
   --role-session-name  <RANDOM_UNIQUE_NAME> \
   --serial-number <ARN_OF_MFA_DEVICE>\
   --token-code  \
    --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' --output text))

# profile used here will be configured in ~/.aws/credentials
export AWS_PROFILE=<PROFILE YOU WANT>

aws configure set aws_access_key_id ${temp_keys[0]}
aws configure set aws_secret_access_key ${temp_keys[1]}
aws configure set aws_session_token ${temp_keys[2]}

```

This creates a temporary valid aws profile, that can be used BOTH with the aws cli, and kubectl.

  • Ok, let me try that. I'm not sure if I completely get it, but I'll check and come back to you, if I have questions :) Thanks! – razr Aug 1 '18 at 9:52
  • Hm. Could not get this to work... I execute the above command with a RANDOM_UNIQUE_NAME I just made up and a TOKEN_VALUE from my MFA device (I just left the MFA_TOKEN_VALUE part out. it was just the var used afterwards, right?) and got a block with credentials (AccessKeyId, SecretAccessKey, SessionToken, Expiration) and an AssumedRoleUser back. aws configure lets me just set the access key id and secret. So what do I do with the other info? And what do I do with the session name? – razr Aug 1 '18 at 10:37
  • So, the end result should be a valid block of credentials in the ~/.aws/credentials file, for the profile you want to use with kubectl, let me edit my answer. – mrArias Aug 2 '18 at 6:34
1

Btw: I managed to fix this problem by using aws-mfa (https://github.com/broamski/aws-mfa), which I have configured to cache my aws credentials for an hour.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.