Docker managers are losing quorum on AWS

Question

I have deployed Docker EE on AWS by choosing the package directly from the AWS Marketplace: Docker EE for AWS (Standard/Advanced) - BYOL. The cluster has been successfully launched and deployed on AWS.

After deploying several stacks on the UCP something really strange is happening, the managers are not able to see each other, they are losing the quorum inside the Swarm and the docker info command says that they are out of Swarm.

The AutoScalling group from AWS starts launching new managers inside the Swarm and the new launched managers are terminated in a couple of minutes because they are not Swarm's participants as well. Actually the AutoScalling group goes in a infinity loop by launching new managers and terminating the old ones.

The worst thing is that I am loosing the data from the Docker Services and there is no way of recovering.

Any ideas?

Answer 1

It seems like the AutoScaling Group from AWS are not that synchronized with the Docker Swarm configuration and the default timeouts in the AutoScaling are not accepted in the Docker Swarm managers.

Actually the Health Check is not configured good by default in the EE CloudFormation template, even if the managers are running and stable the Health Check fails and the AutoScalling Group tries to launch new managers. New launched managers are not able to become cluster participants, right after their running state.