Why does Docker fail to start on AWS ECS instances. ECS stateful instance fails initialization. ECS stateful instance doesn't join the ECS cluster

Question

The ECS instance fails to join the ECS cluster because it relies on docker to launch its ecs agent. If in this case docker fails to start then the whole process halts.

Docker fails to start in AWS ECS instances with the following message in /var/log/docker :

Error starting daemon: error initializing graphdriver: devicemapper: Non 
existing device docker-docker--pool

Notice the following.

Instance AMI is a COPY of the AWS ECS optimized AMI with the embedded ECS agent(and docker config for ECS)

Answer 1

The key here is the behavior of AWS AMIs in the context of ECS. By default the AWS ECS Amazon Linux AMI optimized for ECS is in charge of VOLUME management but there's a problem if you copy that AMI and try to launch a new instance -- the SECONDARY volume that ECS uses to deploy docker on can go missing.

Troubleshooting steps:

• Check /var/log/cloud-init-output.log and see if you have an error message like:

  ERROR: /dev//dev/xvdcz is not a valid block device.

• Check your instance mounted volumes and make sure that the Secondary Volume exists. Usually if you see the error above it means that only /dev/xvda is mounted.

FIX:

Enforce your Launch Configuration, Launch Template or script to always deploy the secondary volume /dev/xvdcz regardless of AMI configuration. Do not rely on the AMI as unfortunately the AMI flow has changed and might not preserve all the necessary configuration.