2

I'm attempting to login to a container (from ECR) deployed into an AWS ECS cluster. For what it's worth I'm using the docker compose/ecs integration to deploy this cluster. My docker compose file is very minimal. The container needs a GPU so I'm deploying it to a GPU instance (g4dn.12xlarge) with an ecs optimized ami, ami-03d0d75de9d82f509 (amzn2-ami-ecs-gpu-hvm-2.0.20221230-x86_64-ebs).

I'm trying to exec into this container but am unable to login.

I've executed this command to attempt login:

aws ecs execute-command --cluster apptest --task 36fd9d835ad24b4ca188e40c59768cee --container apptest --interactive --command "/bin/sh"

I'm getting the following error:

The Session Manager plugin was installed successfully. Use the AWS CLI to start a session.


An error occurred (TargetNotConnectedException) when calling the ExecuteCommand operation: The execute command failed due to an internal error. Try again later.

I would really appreciate any additional info on why I might be getting this error and what I should check/test.

I've run the amazon-ecs-exec-checker script which gives the following output (I have removed some account info). Exec is enabled for the task and I believe all the correct permissions are in place (SSM policy permissions, etc). When I searched for similiar errors on google I saw that this was an issue on older ami's but it should have been fixed.

    Prerequisites for check-ecs-exec.sh v0.7
-------------------------------------------------------------
  jq      | OK (/usr/bin/jq)
  AWS CLI | OK (/usr/local/bin/aws)

-------------------------------------------------------------
Prerequisites for the AWS CLI to use ECS Exec
-------------------------------------------------------------
  AWS CLI Version        | OK (aws-cli/2.9.9 Python/3.9.11 Linux/5.10.149-133.644.amzn2.x86_64 exe/x86_64.amzn.2 prompt/off)
  Session Manager Plugin | OK (1.2.398.0)

-------------------------------------------------------------
Checks on ECS task and other resources
-------------------------------------------------------------
Region : us-east-1
Cluster: apptest
Task   : arn:aws:ecs:us-east-1:****:task/apptest/36fd9d835ad24b4ca188e40c59768cee
-------------------------------------------------------------
  Cluster Configuration  | Audit Logging Not Configured
  Can I ExecuteCommand?  | arn:aws:iam::****:user/***
     ecs:ExecuteCommand: allowed
     ssm:StartSession denied?: allowed
  Task Status            | RUNNING
  Launch Type            | EC2
  ECS Agent Version      | 1.67.2
  Exec Enabled for Task  | OK
  Container-Level Checks |
    ----------
      Managed Agent Status
    ----------
         1. STOPPED (Reason: Received Container Stopped event) for "apptest_ResolvConf_InitContainer" - LastStartedAt: null
         2. RUNNING for "apptest"
    ----------
      Init Process Enabled (apptest-apptest:14)
    ----------
         1. Disabled - "apptest_ResolvConf_InitContainer"
         2. Disabled - "apptest"
    ----------
      Read-Only Root Filesystem (apptest-apptest:14)
    ----------
         1. Disabled - "apptest_ResolvConf_InitContainer"
         2. Disabled - "apptest"
  Task Role Permissions  | arn:aws:iam::****:role/apptest-apptestTaskRole-12KZYNKIW0B65
     ssmmessages:CreateControlChannel: allowed
     ssmmessages:CreateDataChannel: allowed
     ssmmessages:OpenControlChannel: allowed
     ssmmessages:OpenDataChannel: allowed
  VPC Endpoints          | SKIPPED (vpc-020109*** - No additional VPC endpoints required)
  Environment Variables  | (apptest-apptest:14)
       1. container "apptest_ResolvConf_InitContainer"
       - AWS_ACCESS_KEY: not defined
       - AWS_ACCESS_KEY_ID: not defined
       - AWS_SECRET_ACCESS_KEY: not defined
       2. container "apptest"
       - AWS_ACCESS_KEY: not defined
       - AWS_ACCESS_KEY_ID: not defined
       - AWS_SECRET_ACCESS_KEY: not defined

1 Answer 1

1

Check if you can use AWS Session Manager to connect to the instance. I was having the same issue and I found it was because the SSM agent on the instance could not connect due to having restricted egress network access.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.