1

The company I work for wants to prevent provider lock-in adn therefore told us to use Terraform+Ansible instead of Cloudformation. We already decided to use terragrunt as an addon. While the motivation for the company decision is clear they didn't provide guidance how to achieve this and we have to find out on our own how to do this in an "elegant" way.

In my scenario this means I use the two tools like this:

Terraform:

  • provision EC2 instances (Ubuntu 20.04 in this case)
  • define security groups with ingress rules
  • define host name aliases (assigning them is a weak spot, I will come back to this later)

Ansible:

  • install additional OS level components
  • install and configure the application

All of this will be run from a devops machine, after that I have a working application setup. But what happens when for example monitoring decides that one of ten instances is no longer healthy ? That component will then be nuked and created from scratch. In this scenario, how can I make sure that:

  • the replacement instance is configured using the same Ansible playbooks that were run before ?
  • host name aliases will be updated

For the host name alias topic I have to give you a little bit more background on what we need. In one scenario we use an application that does not load balance. Because of this we create for example three instances which we assign aliases:

inst-one   -> lg1.company.foo
inst-two   -> lg2.company.foo
inst-three -> lg3.company.foo

Users access the instances using the aliases

Let's assume that inst-two becomes unhealthy, then this will need to happen:

inst-one   -> lg1.company.foo
inst-two   -> terminated
inst-new   -> lg2.company.foo
inst-three -> lg3.company.foo
3

There are a lot of gaps in your deployment and configuration strategy. In order to provide a more robust and elegant strategy, I suggest you:

  1. Define autoscaling groups with Terraform instead of provisioning individual EC2 instances. Auto-scaling groups make dynamically scaling instances a much simpler process. It will also allow you to define scaling policies and health checks for your instances to determine when to add/remove instances from your autoscaling group.
  2. Create and define CloudWatch Metrics and CloudWatch Alarms. CloudWatch allows you to create custom metrics to monitor and alert via CloudWatch Alarms. This can be used in conjunction with autoscaling group scaling policies to define a threshold to scale your instances based on metrics such as cpu utilization, application response time, etc.
  3. Instead of using Ansible to configure each EC2 instance, use Ansible to configure an Amazon Machine Image (AMI) to be uploaded and used by your autoscaling group. If your instances are configured identically, configuration is a one and done process. If your instances are configured differently, consider uploading multiple AMIs, and separating into multiple autoscaling groups. You can also use the User Data definition of an EC2 instance to run shell commands on each individual instance at launch (a one time process).
  4. Last, but not least, instead of running this process from "a devops machine", consider using a build/deploy CI pipeline to run and configure this entire process. Tools such as AWS CodePipeline, Jenkins, GitLab CI, etc. allow for you and other team members to run your configuration in a configured and repeatable manner. It will also allow you define stages and deployment strategies should you choose to implement testing for your application later on in your development process.
4
  • 1
    I like this answer, so upvoted. May I suggest adding some preamble about states being declarative? IE, "make all the cloud infra in terraform and discover the AMI from tags, but make the AMI before planning the infra, upstream in the pipeline. Since userdata changes at every plan, keeping the machine state declarative using Ansible will make changes easier to see in CI and easier to manage" – Bruce Becker Dec 18 '20 at 5:42
  • Thank you. While your advice is certainly good (especially #3 and #4) auto scaling groups are not what I'm looking for. We need to provision an exact amount of servers and each of them needs to have a dedicate hostname which is defined in route53. How can I make sure that each of the aliases is assiged to a "living" instance ? – Marged Dec 19 '20 at 14:31
  • @Marged I would be very curious as to what your use case for having an "exact" amount of servers is. It sounds like your team is giving you direction to migrate/develop this application in the cloud without understanding the implications of running a cloud-native app. Even if you need an exact amount of servers, you can certainly declare multiple autoscaling groups of size 1 as a way to replace "unhealthy" instances. If you need route53 aliases assigned to any of these instances, place your ASG behind an ELB, then point route53 to your ELB. – Preston Martin Dec 20 '20 at 22:32
  • @PrestonMartin We are running load tests and know which amount of servers we need to create a certain amount of "pressure" on the target systems. We provision them when the load test starts and kill them directly afterwards. In the loadtest project we have a section where we define the servers to use. And we want to avoid that every time we create new servers the endusers have to update their loadtest project. Because of this we want to have the aliases. Load balancing doesn't make sense there, the loadtest tool addresses the servers (=loadgenerators) on its own. – Marged Dec 21 '20 at 13:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.