Run integration test with ansible playbook using terraform provisioning

Question

Ok so I am totally new to devops! I am an experienced developer, but at my new job I have been tasked to automate CI/CD. Non-existing so far. Only CI with Travis.

So the ideal workflow is: Push to CI -> build -> if successful provision an EC2 instance -> run integration tests

Our ideal architecture is to use terraform for provisioning and ansible for automation/config management. Scenarios down the road foresee continuous deployment and more stuff. So I need to get the basic building blocks right. This workflow seems like a good start.

I've searched for tutorials all day and, being beginner, I am totally confused.

As the goal is to fully automate the process, I am naively assuming that the ansible playbook would call a terraform provisioning role/task, and then continue by running all the steps for the integration tests.

But all I have found says to always run terraform init first and then terraform apply before switching to run a playbook, which seems cumbersome and not that "automated".

Can someone point me to a good resource describing how I can setup my desired workflow? Am I understanding things wrong?

Answer 1

You are right about Terraform having a plan-apply cycle which may not be that conducive to automation. The following has worked for me in similar situations:

1. Use Terraform to "develop" new versions of a base EC2 image.
2. Use Ansible playbooks to provision EC2 instances off a specific version of the image and run tests on them.

Answer 2

This is indeed a tricky thing to do "right", because there are so many moving parts. I have found that it helps to reduce the degrees of freedom by adopting some conventions, and factoring out some parts.

Draw a line between where Terraform is responsible and where Ansible becomes responsible, and separate those stages. You do this so that you can deal with smaller bits and verify the changes.

This way, you can maintain smaller modules of the pipeline, and create deployment or testing scenarios from various well-known starting points.

The next strategy that might help is to "start from the end" and automate the rest. Ie, write a set of assertions on the final state of your pipeline (we do this with Inspec profiles. In this way, you can know whether the application of your Terraform and Ansible has had the desired effect (ie, you can "fail the build"), and roll back.

It's not clear from your post what the complexity of your inventory is, but if it's something like a few machines deployed in a cloud, you could do the following:

1. Terraform the cloud
   1. terraform init
   2. terraform init
   3. terraform plan -out $BUILD_NUMBER
   4. terraform apply -auto-approve $BUILD_NUMBER
   5. (ensure that you have an output.tf to generate an inventory to target for Ansible in the next steps).
2. Assert correct state of the infrastructure:
   1. inspec exec cloud_profile
3. Configure instances
   1. `ansible-playbook -i playbook.yml
4. Assert the correct state of the converged instances:
   1. inspec exec instance-profile

Now you have known-good infrastructure and machines, resulting from an automated pipeline an what is more, you will know clearly (from where the pipeline fails) if there is a problem.

Final thought - you could go one step further in separating the infrastructure from configuration management by building and testing your images in a separate pipeline, with a Packer template. This could use the same Ansible playbook and Inspec profile to generate a known-good image which you can then pass to Terraform at apply time.

The feasibility of these approaches depends on how interconnected the resources are in your setup. It might help to draw an abstract dependency graph of the pipeline steps to map actions and tools to stages.

Answer 3

This scenario remembers me how i used to test my ansible playbook with terraform. I would run "remote exec" in terraform to install python and other dependencies of ansible. After that I would retrieve the public ip of the instance and save in the inventory file of ansible in the local. And finally, I ran ansible from my local machine from terraform configuration using "local exec". Of course, before everything you need to configure SSH from local to newly provisioned machine in terraform configuration.

resource "null_resource" "ConfigureAnsibleLabelVariable" {
  provisioner "local-exec" {
    command = "echo [${var.dev_host_label}:vars] > hosts"
  }
  provisioner "local-exec" {
    command = "echo ansible_ssh_user=${var.ssh_user_name} >> hosts"
  }
  provisioner "local-exec" {
    command = "echo ansible_ssh_private_key_file=${var.ssh_key_path} >> hosts"
  }
  provisioner "local-exec" {
    command = "echo [${var.dev_host_label}] >> hosts"
  }
}

resource "null_resource" "ProvisionRemoteHostsIpToAnsibleHosts" {
  count = "${var.instance_count}"
  connection {
    type = "ssh"
    user = "${var.ssh_user_name}"
    host = "${element(aws_instance.DevInstanceAWS.*.public_ip, count.index)}"
    private_key = "${file("${var.ssh_key_path}")}"
  }
  provisioner "remote-exec" {
    inline = [
      "sudo yum update -y",
      "sudo yum install python-setuptools python-pip -y",
      "sudo pip install httplib2"
    ]
  }
  provisioner "local-exec" {
    command = "echo ${element(aws_instance.DevInstanceAWS.*.public_ip, count.index)} >> hosts"
  }
}

resource "null_resource" "ModifyApplyAnsiblePlayBook" {
  provisioner "local-exec" {
    command = "sed -i -e '/hosts:/ s/: .*/: ${var.dev_host_label}/' play.yml"   #change host label in playbook dynamically
  }

  provisioner "local-exec" {
    command = "sleep 10; ansible-playbook -i hosts play.yml"
  }
  depends_on = ["null_resource.ProvisionRemoteHostsIpToAnsibleHosts"]
}