Using a combination of Terraform and Ansible to create dynamic AWS EC2 infrastructure

Question

The company I work for wants to prevent provider lock-in adn therefore told us to use Terraform+Ansible instead of Cloudformation. We already decided to use terragrunt as an addon. While the motivation for the company decision is clear they didn't provide guidance how to achieve this and we have to find out on our own how to do this in an "elegant" way.

In my scenario this means I use the two tools like this:

Terraform:

• provision EC2 instances (Ubuntu 20.04 in this case)
• define security groups with ingress rules
• define host name aliases (assigning them is a weak spot, I will come back to this later)

Ansible:

• install additional OS level components
• install and configure the application

All of this will be run from a devops machine, after that I have a working application setup. But what happens when for example monitoring decides that one of ten instances is no longer healthy ? That component will then be nuked and created from scratch. In this scenario, how can I make sure that:

• the replacement instance is configured using the same Ansible playbooks that were run before ?
• host name aliases will be updated

For the host name alias topic I have to give you a little bit more background on what we need. In one scenario we use an application that does not load balance. Because of this we create for example three instances which we assign aliases:

inst-one   -> lg1.company.foo
inst-two   -> lg2.company.foo
inst-three -> lg3.company.foo

Users access the instances using the aliases

Let's assume that inst-two becomes unhealthy, then this will need to happen:

inst-one   -> lg1.company.foo
inst-two   -> terminated
inst-new   -> lg2.company.foo
inst-three -> lg3.company.foo

Answer 1

There are a lot of gaps in your deployment and configuration strategy. In order to provide a more robust and elegant strategy, I suggest you:

1. Define autoscaling groups with Terraform instead of provisioning individual EC2 instances. Auto-scaling groups make dynamically scaling instances a much simpler process. It will also allow you to define scaling policies and health checks for your instances to determine when to add/remove instances from your autoscaling group.
2. Create and define CloudWatch Metrics and CloudWatch Alarms. CloudWatch allows you to create custom metrics to monitor and alert via CloudWatch Alarms. This can be used in conjunction with autoscaling group scaling policies to define a threshold to scale your instances based on metrics such as cpu utilization, application response time, etc.
3. Instead of using Ansible to configure each EC2 instance, use Ansible to configure an Amazon Machine Image (AMI) to be uploaded and used by your autoscaling group. If your instances are configured identically, configuration is a one and done process. If your instances are configured differently, consider uploading multiple AMIs, and separating into multiple autoscaling groups. You can also use the User Data definition of an EC2 instance to run shell commands on each individual instance at launch (a one time process).
4. Last, but not least, instead of running this process from "a devops machine", consider using a build/deploy CI pipeline to run and configure this entire process. Tools such as AWS CodePipeline, Jenkins, GitLab CI, etc. allow for you and other team members to run your configuration in a configured and repeatable manner. It will also allow you define stages and deployment strategies should you choose to implement testing for your application later on in your development process.