How to support releasing new versions of the code, running in parallel with the last stable release?

Question

I have one service in production (on AWS), which follows the immutable server pattern. Its deployment looks like this:

1. Create a new AMI with Packer.
2. Create a new CloudFormation stack, starting with an auto-scaling group of size 1.

When I see that the new release is good, I can increase the number of instances and eventually shutdown old instances and finally remove the CloudFormation stack from the previous release.

In my initial version of the deployment, I used only one stack and updated it in place. For normal releases, that meant an the auto-scaling group was modified by CloudFormation to point to the new AMI. Then I had to either kill an existing instance or increase the auto scaling group to get an instance with the release running.

Using a new stack for every release makes the process simpler for me, as rollbacks are easier and it makes rolling out releases to parts of the users easier. Similar to the immutable server pattern, I avoid in-place updates but instead only create new resources (stacks in this case).

In the company that I work for, it is now more common to use Terraform instead of CloudFormation. I wonder if it is possible to adept the deployment that I sketched with Terraform. I don't mind using another tool, my main point is that I would like to preserve these basic concepts:

• Allow deployments of new releases without touching the stable release
• Instead of (in-place) updating the setup, only create new resources and kill the old resources.

So, far I have only briefly worked with Terraform and used it only to manage some small part for our infrastructure. As recommended, I kept the state in an S3 bucket, for instance:

# (from main.tf)
terraform {
  required_version = ">= 0.9.4"
  backend "s3" {
    bucket = "example-company-terraform-state"
    key    = "/foo-service/terraform.tfstate"
    region = "eu-central-1"
  }
}

Here, the key is always fixed. So, Terraform will update everything in place. I assume you could use a new key for new releases to have a similar effect as creating new CloudFormation stacks.

What I did not figure out is how to move traffic to the new instances. All instances should be behind an load balancer (ELB). I think, you could use a separated setup for the ELB, which will distribute traffic to instances from both the old and the new release.

So, if a new release is rolled out, there would be three different Terraform states:

1. Terraform setup for the ELB
2. Terraform setup for the old release (more specifically, its auto-scaling group)
3. Terraform setup for the new release

Questions:

• Has the approach of using different S3 keys for the Terraform state the same effect as using separate CloudFormation stacks? (What I'm looking for is a way to have multiple Terraform setups that do not interfere with each other.)
• Do you see a solution to the problem of getting all instances (from the old and the new release) behind a shared ELB?
• I read that you can export resources in Terraform. Do you think, I could create a Terraform setup for the ELB, export the ELB as a resource, and use it in the Terraform setup for the instances (plus auto-scaling group) to make them attach to the shared ELB?

---

Note: For my other service, where I use CloudFormation, we do not use an ELB, so it is strictly speaking a different problem. I only mentioned it to explain what I found useful about the deployment and why I think about applying some of the ideas for other services.

Answer 1

There are a few different ways to achieve goals of this sort, each with some different tradeoffs. I'm going to describe the most common ones below.

---

The simplest approach is to use Terraform's create_before_destroy mechanism with autoscaling groups. An example of this pattern is included in the aws_launch_configuration documentation.

In this scenario, changing the AMI id causes the launch configuration to be re-created. Due to create_before_destroy, the new configuration is created first, then a new autoscaling group is created, adding the new instances to an attached ELB. The min_elb_capacity argument to aws_autoscaling_group can be used to ensure that a given number of instances are present and healthy in the attached ELB before considering the autoscaling group to be created, thus delaying the destruction of the old autoscaling group and launch configuration until the new one is serving requests.

The downside of this approach is the lack of control it represents. Since Terraform is thinking of the entire set of changes as a single run, it's impossible to pause after creating the new instances to allow other checks to be carried out before destroying the old ones. As a consequence, the ELB healthcheck is the only input to deciding if the new release is "good", and rolling back is impossible once the old resources have been destroyed.

---

A second common approach is to adopt a sort of "blue/green deployment" pattern with explicit changes to two clusters. This is done by putting all of the per-release resources in a child module, and instantiating that module twice with different arguments. In the top-level module this would look something like the following:

resource "aws_elb" "example" {
  instances = "${concat(module.blue.ec2_instance_ids, module.green.ec2_instance_ids)}"

  # ...
}

module "blue" {
  source = "./app"

  ami_id = "ami-1234"
  count  = 10
}

module "green" {
  source = "./app"

  ami_id = "ami-5678"
  count  = 0
}

The principle of operation here is that in the "steady state" (no deployment in progress) only one of these modules has a non-zero count, and the other one has zero. During a deployment, they are both set to the same non-zero count, but with different ami_id values. Each deployment swaps which of the modules is the "active" module, with both being active during the deployment.

When using this approach, each step is a distinct Terraform operation:

1. change count of the inactive module to nonzero and set its AMI id
2. apply the change with Terraform, thus activating the new module
3. verify that the new release is good
4. change count of the older module to zero
5. apply the change with Terraform, thus deactivating the old module

Although this has more steps, it allows arbitrary verification and an arbitrary amount of time to pass during step 3. It also allows "rolling back" by resetting the previously-inactive cluster count to zero.

Since both the old and new clusters exist in the same configuration, there is the risk of using this pattern incorrectly and prematurely destroying the active cluster. This can be mitigated by carefully reviewing Terraform's plan to make sure it leaves the old cluster untouched, but Terraform itself can't guarantee this.

Also, since both clusters are using the same child module configuration, it can be tricky to make updates to that configuration while retaining the blue/green separation. If changes are made that would require Terraform to replace the running instances, it's necessary to temporarily have two copies of the module code on disk, make the source arguments point to separate copies, and make the change only to the copy used by the inactive module.

---

The final approach I'll present is the most extreme and manual, but it does the best job of meeting your requirements and retaining control. This is, in effect, the most literal interpretation of your current CloudFormation workflow, and is a more concrete version of the approach you talked about in your question.

In this approach, there are two entirely-separate Terraform configurations, which I will call "version-agnostic" (things that must survive between versions, such as your ELB) and "version-specific" (the resources that are re-created for each new version).

The version-agnostic configuration will contain the ELB and will, as you suspected, export its id for consumption by the version-specific configuration:

terraform {
  required_version = ">= 0.9.4"
  backend "s3" {
    bucket = "example-company-terraform-state"
    key    = "exampleapp/version-agnostic"
    region = "eu-central-1"
  }
}

resource "aws_elb" "example" {
  # ...
}

output "elb_id" {
  value = "${aws_elb.example.id}"
}

This configuration can be initialized, planned and applied as usual, creating an ELB with no attached instances to start.

The version-specific configuration would be similar to the "app" child module in the previous approach, but this time as a top-level module. The backend configuration for this module would omit the S3 key, since this will change for each new release as you expected:

terraform {
  required_version = ">= 0.9.4"
  backend "s3" {
    bucket = "example-company-terraform-state"
    region = "eu-central-1"
  }
}

The specific key can then be set (or re-set) when running terraform init:

$ terraform init -reconfigure -backend-config="key=exampleapp/20170808-1"

Here I chose to use a "current date, release index" tuple as an identifier for a release. By running terraform init with a new value for this argument, an entirely separate state is created, independent of the last. Using -reconfigure tells Terraform that you don't wish to migrate the old state to the new, but rather to just switch directly to the new state path, possibly creating a new state in the process.

You can then run terraform show to confirm that indeed the state is empty (and thus operations won't affect existing resources) and then run a plan/apply cycle as normal.

Once you're satisfied with the new release, you can switch back to the previous version and destroy it.

The version-specific configuration will need the id of the ELB from the version-agnostic configuration in order to populate the load_balancers attribute of aws_autoscaling_group. To get access to this, we can use the terraform_remote_state data source to read the values from its state in S3:

data "terraform_remote_state" "version_agnostic" {
  backend = "s3"
  config {
    bucket = "example-company-terraform-state"
    key    = "exampleapp/version-agnostic"
    region = "eu-central-1"
  }
}

resource "aws_autoscaling_group" "example" {
  # ...

  load_balancers = ["${data.terraform_remote_state.version_agnostic.elb_id}"]
}

With a system of this complexity, it would likely be best to run Terraform via some sort of wrapper script or orchestration to make the release process less onerous. For example, such a script might automatically generate the new version number to avoid the risk of a human operator mistyping the date or accidentally conflicting with an existing one. There's some recommendations and caveats about running Terraform via scripts in the guide Running Terraform in Automation.

---

Although the third option here is the most direct mapping of your CloudFormation approach, the second option is more commonly used due to it striking a reasonable compromise between control and workflow overhead.