8

I need to do some large scale testing on a system. The system consists of 2 main parts: Manager and Nodes. There will be only one manager server, but 100 nodes. Both the manager and nodes are Docker images.

I want to be able to spin up a setup on AWS, where an instance of a manager, and 100 instances (one instance per server) of the node is started.

What would I use for this? The manager and nodes should be able to communicate together, and the nodes should not be load balanced in any way. So they need a dedicated internal IP that the manager can communicate with.

I've looked at ECS and CloudFormation+EC2 with a custom AMI, but I'm not sure what is the way to go.

Ideally I want a script that I once in a while can run to launch the servers, and close them down easily.

Any suggestions on what I can do on AWS to launch this setup with ease?

Improve this question

2 Answers 2

Reset to default
5

Look at AWS Fargate - it lets you run your Docker containers without having to spin up the EC2 instances for an ECS cluster. You simply schedule the Tasks (or Services), each gets is own IP address and they can talk to each other.

You can also use AWS Service Discovery to facilitate the registration and lookup of the tasks' IP addresses.

BTW Note that the default Fargate concurrency limit is 20 Tasks/Services - if you need 100 of them you'll have to raise a Service Limit increase support ticket.

We've got it raised to 200 concurrent tasks because our processing runs 160+ containers in parallel, all on Fargate without a single EC2 to manage.

Update - how to spin up 100 containers

I can tell you how we do it with Fargate - in our CI/CD pipeline we build the container image and upload to ECR. Part of the source is also a CloudFormation (CFN) template and in the Deployment stage of the pipeline we create/update CFN stack which creates ECS TaskDefinition with all the container parameters.

Then we've got a scheduler job written in Python and triggered by CloudWatch Event (cron-like scheduler) that looks up the TaskDefinition ARN and runs 160+ Tasks from that single TaskDef. You can give each Task an extra parameter, or override some config, etc. Or not and just spin it up.

We used to use EC2/ECS and had to do an extra step first - scale out the ECS AutoScaling group to tens of nodes to support our workload and then at the end scale it back in to 0. Other than that it's the same whether you run it on Fargate or on EC2/ECS.

Hope that helps :)

Improve this answer
3
  • I looked at Fargate too, but the nodes are running simulations, so I'm not sure if it's "bad" that I cannot control the instance size of them. But isn't the initial setup of ECS/EC2 vs. ECS/Fargate pretty much the same, other than you don't have to think about EC2's?
    – Trolley
    Sep 27, 2018 at 10:05
  • @Trolley updated the answer with our method of spinning up the 160+ containers.
    – MLu
    Sep 28, 2018 at 23:46
  • @Trolley Fargate reserves CPU and Memory for your containers - we didn't observe any degradation in performance on Fargate compared to EC2.
    – MLu
    Sep 28, 2018 at 23:48
2

Why don't you use Terraform for the purpose? It can launch as many instances you want, create network for you, create security groups, ECS, IAMs or whatever you want to create and terminate easily when you want. This is widely used in industry now.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.