I cannot exec into a docker container running in ECS

Question

I'm attempting to login to a container (from ECR) deployed into an AWS ECS cluster. For what it's worth I'm using the docker compose/ecs integration to deploy this cluster. My docker compose file is very minimal. The container needs a GPU so I'm deploying it to a GPU instance (g4dn.12xlarge) with an ecs optimized ami, ami-03d0d75de9d82f509 (amzn2-ami-ecs-gpu-hvm-2.0.20221230-x86_64-ebs).

I'm trying to exec into this container but am unable to login.

I've executed this command to attempt login:

aws ecs execute-command --cluster apptest --task 36fd9d835ad24b4ca188e40c59768cee --container apptest --interactive --command "/bin/sh"

I'm getting the following error:

The Session Manager plugin was installed successfully. Use the AWS CLI to start a session.


An error occurred (TargetNotConnectedException) when calling the ExecuteCommand operation: The execute command failed due to an internal error. Try again later.

I would really appreciate any additional info on why I might be getting this error and what I should check/test.

I've run the amazon-ecs-exec-checker script which gives the following output (I have removed some account info). Exec is enabled for the task and I believe all the correct permissions are in place (SSM policy permissions, etc). When I searched for similiar errors on google I saw that this was an issue on older ami's but it should have been fixed.

    Prerequisites for check-ecs-exec.sh v0.7
-------------------------------------------------------------
  jq      | OK (/usr/bin/jq)
  AWS CLI | OK (/usr/local/bin/aws)

-------------------------------------------------------------
Prerequisites for the AWS CLI to use ECS Exec
-------------------------------------------------------------
  AWS CLI Version        | OK (aws-cli/2.9.9 Python/3.9.11 Linux/5.10.149-133.644.amzn2.x86_64 exe/x86_64.amzn.2 prompt/off)
  Session Manager Plugin | OK (1.2.398.0)

-------------------------------------------------------------
Checks on ECS task and other resources
-------------------------------------------------------------
Region : us-east-1
Cluster: apptest
Task   : arn:aws:ecs:us-east-1:****:task/apptest/36fd9d835ad24b4ca188e40c59768cee
-------------------------------------------------------------
  Cluster Configuration  | Audit Logging Not Configured
  Can I ExecuteCommand?  | arn:aws:iam::****:user/***
     ecs:ExecuteCommand: allowed
     ssm:StartSession denied?: allowed
  Task Status            | RUNNING
  Launch Type            | EC2
  ECS Agent Version      | 1.67.2
  Exec Enabled for Task  | OK
  Container-Level Checks |
    ----------
      Managed Agent Status
    ----------
         1. STOPPED (Reason: Received Container Stopped event) for "apptest_ResolvConf_InitContainer" - LastStartedAt: null
         2. RUNNING for "apptest"
    ----------
      Init Process Enabled (apptest-apptest:14)
    ----------
         1. Disabled - "apptest_ResolvConf_InitContainer"
         2. Disabled - "apptest"
    ----------
      Read-Only Root Filesystem (apptest-apptest:14)
    ----------
         1. Disabled - "apptest_ResolvConf_InitContainer"
         2. Disabled - "apptest"
  Task Role Permissions  | arn:aws:iam::****:role/apptest-apptestTaskRole-12KZYNKIW0B65
     ssmmessages:CreateControlChannel: allowed
     ssmmessages:CreateDataChannel: allowed
     ssmmessages:OpenControlChannel: allowed
     ssmmessages:OpenDataChannel: allowed
  VPC Endpoints          | SKIPPED (vpc-020109*** - No additional VPC endpoints required)
  Environment Variables  | (apptest-apptest:14)
       1. container "apptest_ResolvConf_InitContainer"
       - AWS_ACCESS_KEY: not defined
       - AWS_ACCESS_KEY_ID: not defined
       - AWS_SECRET_ACCESS_KEY: not defined
       2. container "apptest"
       - AWS_ACCESS_KEY: not defined
       - AWS_ACCESS_KEY_ID: not defined
       - AWS_SECRET_ACCESS_KEY: not defined

Answer 1

Check if you can use AWS Session Manager to connect to the instance. I was having the same issue and I found it was because the SSM agent on the instance could not connect due to having restricted egress network access.