Best way to launch 100 identical Docker machines on AWS?

Question

I need to do some large scale testing on a system. The system consists of 2 main parts: Manager and Nodes. There will be only one manager server, but 100 nodes. Both the manager and nodes are Docker images.

I want to be able to spin up a setup on AWS, where an instance of a manager, and 100 instances (one instance per server) of the node is started.

What would I use for this? The manager and nodes should be able to communicate together, and the nodes should not be load balanced in any way. So they need a dedicated internal IP that the manager can communicate with.

I've looked at ECS and CloudFormation+EC2 with a custom AMI, but I'm not sure what is the way to go.

Ideally I want a script that I once in a while can run to launch the servers, and close them down easily.

Any suggestions on what I can do on AWS to launch this setup with ease?

Answer 1

Look at AWS Fargate - it lets you run your Docker containers without having to spin up the EC2 instances for an ECS cluster. You simply schedule the Tasks (or Services), each gets is own IP address and they can talk to each other.

You can also use AWS Service Discovery to facilitate the registration and lookup of the tasks' IP addresses.

BTW Note that the default Fargate concurrency limit is 20 Tasks/Services - if you need 100 of them you'll have to raise a Service Limit increase support ticket.

We've got it raised to 200 concurrent tasks because our processing runs 160+ containers in parallel, all on Fargate without a single EC2 to manage.

Update - how to spin up 100 containers

I can tell you how we do it with Fargate - in our CI/CD pipeline we build the container image and upload to ECR. Part of the source is also a CloudFormation (CFN) template and in the Deployment stage of the pipeline we create/update CFN stack which creates ECS TaskDefinition with all the container parameters.

Then we've got a scheduler job written in Python and triggered by CloudWatch Event (cron-like scheduler) that looks up the TaskDefinition ARN and runs 160+ Tasks from that single TaskDef. You can give each Task an extra parameter, or override some config, etc. Or not and just spin it up.

We used to use EC2/ECS and had to do an extra step first - scale out the ECS AutoScaling group to tens of nodes to support our workload and then at the end scale it back in to 0. Other than that it's the same whether you run it on Fargate or on EC2/ECS.

Hope that helps :)

Answer 2

Why don't you use Terraform for the purpose? It can launch as many instances you want, create network for you, create security groups, ECS, IAMs or whatever you want to create and terminate easily when you want. This is widely used in industry now.