How to test a Terraform Configuration?

Question

If you had a Terraform configuration that had a moderate degree of complexity, how would you write tests around the configuration that could be executed as part of a Continuous Integration / Continuous Delivery pipeline?

As an example, you may have a multi-cloud configuration that specifies the following desired state:

• Azure Container Services to host Docker in Azure
• Azure Blob Storage
• SQL Azure
• EC2 Container Service to host Docker in AWS
• Amazon S3 Storage Service
• Amazon RDS SQL Server database

Potentially terraform apply could create the above from scratch, or transition from a partially deployed state to the above desired state.

I am aware that Terraform splits its work into the execution plan stage and the application phase which actually makes changes to the target architecture. Can this be used to write tests against the execution plan, if so are there frameworks to help write these?

Answer 1

There is currently no full solution to this integrated into Terraform, but there are some building blocks that could be useful to assist in writing tests in a separate programming language.

Terraform produces state files in JSON format that can, in principle, be used by external programs to extract certain data about what Terraform created. While this format is not yet considered officially stable, in practice it changes infrequently enough that people have successfully integrated with it, accepting that they might need to make adjustments as they upgrade Terraform.

What strategy is appropriate here will depend a lot on what exactly you want to test. For example:

• In an environment that's spinning up virtual servers, tools like Serverspec can be used to run tests from the perspective of these servers. This can either be run separately from Terraform using some out-of-band process, or as part of the Terraform apply using the remote-exec provisioner. This allows verification of questions like "can the server reach the database?", but is not suitable for questions such as "is the instance's security group restrictive enough?", since robustly checking that requires accessing data from outside of the instance itself.

• It's possible to write tests using an existing test framework (such as RSpec for Ruby, unittest for Python, etc) which gather relevant resource ids or addresses from the Terraform state file and then use the relevant platform's SDK to retrieve data about the resources and assert that they are set up as expected. This is a more general form of the previous idea, running the tests from the perspective of a host outside of the infrastructure under test, and can thus collect a broader set of data to make assertions on.

• For more modest needs, one can choose to trust that the Terraform state is an accurate representation of reality (a valid assumption in many cases) and simply assert directly on that. This is most appropriate for simple "lint-like" cases, such as verifying that the correct resource tagging scheme is being followed for cost-allocation purposes.

There is some more discussion about this in a relevant Terraform Github issue.

In the latest versions of Terraform it is strongly recommended to use a remote backend for any non-toy application, but that means that the state data is not directly available on local disk. However, a snapshot of it can be retrieved from the remote backend using the terraform state pull command, which prints the JSON-formatted state data to stdout so it can be captured and parsed by a calling program.

Answer 2

We recently open sourced Terratest, our swiss army knife for testing infrastructure code.

Today, you're probably testing all your infrastructure code manually by deploying, validating, and undeploying. Terratest helps you automate this process:

1. Write tests in Go.
2. Use helpers in Terratest to execute your real IaC tools (e.g., Terraform, Packer, etc.) to deploy real infrastructure (e.g., servers) in a real environment (e.g., AWS).
3. Use helpers in Terratest to validate that the infrastructure works correctly in that environment by making HTTP requests, API calls, SSH connections, etc.
4. Use helpers in Terratest to undeploy everything at the end of the test.

Here's an example test for some Terraform code:

terraformOptions := &terraform.Options {
  // The path to where your Terraform code is located
  TerraformDir: "../examples/terraform-basic-example",
}

// This will run `terraform init` and `terraform apply` and fail the test if there are any errors
terraform.InitAndApply(t, terraformOptions)

// At the end of the test, run `terraform destroy` to clean up any resources that were created
defer terraform.Destroy(t, terraformOptions)

// Run `terraform output` to get the value of an output variable
instanceUrl := terraform.Output(t, terraformOptions, "instance_url")

// Verify that we get back a 200 OK with the expected text
// It can take a minute or so for the Instance to boot up, so retry a few times
expected := "Hello, World"
maxRetries := 15
timeBetweenRetries := 5 * time.Second
http_helper.HttpGetWithRetry(t, instanceUrl, 200, expected, maxRetries, timeBetweenRetries)

These are integration tests, and depending on what you're testing, can take 5 - 50 minutes. It's not fast (though using Docker and test stages, you can speed some things up), and you'll have to work to make the tests reliable, but it is well worth the time.

Check out the Terratest repo for docs and lots of examples of various types of infrastructure code and the corresponding tests for them.

Answer 3

As an update to this question, there is now Kitchen-Terraform which allows the testing of Terraform Configuration files without breaking production environments. The repository also includes a few examples for different Terraform providers.

Answer 4

In addition to all the other options mentioned, I would like to mention that InSpec 2.0 added support for cloud provider APIs. Basically, you can continue writing the IaC with Terraform, then write compliancy checks with InSpec for your cloud resources. In addition, InSpec supports writing tests for individual machines if you ever need it.

Here is an article from Christoph Hartmann (co-creator of Inspec) on how to use Inspec with Terraform: https://lollyrock.com/articles/inspec-terraform/

Answer 5

On Aws-Side there is https://github.com/k1LoW/awspec - it should be possible, to feed in terraform.state and test, wheter terraform applied correct.

But I think, beyond testing on low level the tool, you used, it's probably a better idea, to think about how to test whole infrastructures.

We're discussing around this idea here:

https://github.com/DomainDrivenArchitecture/dda-cloudspec/blob/development/README.md

For testing invariants upfront, I do not know a ready to use solution ...

We did some experiments using a mix of terraform plan -out=plan.dump and grep for the absence of element names. There is a discussion about a more accessible plan format here: github.com/hashicorp/terraform/issues/11883

But at the moment we're using a manual plan review process for important parts of our infrastructure.

Answer 6

I saw this elegant, low tech method to test Terraform suggested by apparentlymart in a GitHub issue thread. It's not a appropriate for every situation but it's great for verifying module logic.

Create a root module that includes the module under test and verifies the under-test outputs. Here is a simple example using two files:

• main.tf that will run the tests
• simple_module/outputs.tf that represents a module under test

./main.tf

terraform {
  required_version = ">= 0.12"
}

module "simple_module" {
  source = "./simple_module"
}

locals {
  expected = 1
  got      = module.simple_module.module-returns-1
}

# Test Output
output "expect-1" {
  value = upper(local.expected == local.got)
}

output "expect-other" {
  value = "other" == local.got ? upper(true) : "FALSE. Got ${local.got}"
}

./simple_module/outputs.tf

output "module-returns-1" {
  value = 1
}

Run the tests

terraform init
terraform apply -auto-approve

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

expect-1 = TRUE
expect-other = FALSE. Got 1