How to test provisioning and configuration in Ansible setup?

Question

Looking at trying to build some resilience into our Ansible setup which deals with provisioning and configuration.

I understand a few methods of testing on the configuration side of things but I'm wondering how best to implement testing on the provisioning side of things, and if there are any tools which can help with this type of implementation.

Currently a lot of our testing is done serially during the playbook which makes a lot of sense for stuff like "has service come up; is the vip available; has this async task finished" but what really concerns me is our ability to manage drift of configuration at both the application and provisioning layer (such as VM configuration). I'm aware Ansible isn't the best tool for working with configuration drift but I'm curious to see your own opinions.

If you have something to fully automate the process even better. (we have a few ugly scripts which report back in slack daily).

Note: Right now we have a few conditions where a reprovision might occur (e.g. rebuild from backup, critical systems issue) but typically it just loops through some of the ansible configuring tasks and thinks no more of it.

Answer 1

Some options out there..

Testing tools: Sorted by github stars

• Serverspec - Ruby, most popular tool out there, built on ruby's rspec
• Goss - YAML, simple, <10MB self-contained binary, extremely fast, can generate tests from system state
• Inspec - Ruby, think of it as an improved serverspec, almost same syntax, made by the chef guys. Built to be easier to extend than serverspec
• Testinfra - Python, has the cool feature of being able to use Ansible's inventory/vars

Major differences between them:

Ultimately, I would suggest spending a day experimenting with all of them to get a feel for them before deciding for yourself.

• With the exception of Goss, all the frameworks can run against a remote machine (ex. over ssh). Goss only runs locally or in docker w/ dgoss.
• All frameworks can be run locally on a server, but require Python or Ruby to be installed or embedded. Inspec provides a self-contained <150MB bundle with an embedded version of ruby. Goss is a single <10MB binary with no external dependencies.
• Goss has built in support for nagios/sensu output, this allows for easier integration with monitoring tools.
• Goss tests tend to be simpler, but less flexible since it's based on YAML. Other frameworks allow you to leverage the full power of the underlying language Python/Ruby to write tests or extend the tool's functionality. (simplicity vs flexibility)
• Goss allows you to generate tests from current system state
• Testinfra to my knowledge is the only one that has built-in support for ansible inventory and variables
• Inspec is backed by Chef

Continuous/divergence testing:

• Chef Compliance - works with inspec to continuously test your servers, paid product
• Goss - Can be easily hooked into Nagios or Sensu. Also, supports exposing server tests as an http endpoint.

Testing harnesses for development:

• kitchen - Testing harness tool, launches instance, runs config management code, runs test suite. Made by the chef guys
• Molecule - Similar to test kitchen, but written specifically for ansible

Full Disclosure: I'm the author of goss

UPDATE: InSpec 4.x or above uses a mixed commercial / open source license - see comments.

Answer 2

The two tools I've seen for this are InSpec and ServerSpec. Serverspec is a Ruby-based tool that builds on RSpec. InSpec is inspired by RSpec and ServerSpec.

I've used ServerSpec. It's cool, but maybe not 100% stable. I've had problems with testing for specific versions of software on Ubuntu.

I've read the InSpec docs but haven't dug in deep. It does essentially the same thing as Serverspec.

Judging by the Github commits, it looks like work on ServerSpec has tailed off somewhat, whereas InSpec is just getting ramped up.

UPDATE: InSpec 4.x or above uses a mixed commercial / open source license - see comments.

Answer 3

When using configuration management tools, such as Ansible, the tool itself would be responsible preventing configuration drift. Once you used Ansible to set a certain configuration, repetitive execution of Ansible will ensure your configuration is as you defined it to be. This also requires your Ansible code to be written in a manner which is idempotent.

Given the above, testing provisioning can be achieved by executing your Ansible playbooks in a loop from some server. For example, a cron job, or Jenkins, can execute the playbooks every 30 minutes and report back on any failure. Having no failures means that your configuration is in check, having failures means that there is a problem getting servers into your desired state.

In a case where you cannot trust your code to be written as idempotent, and thus you cannot really execute Ansible repetitively in a loop from an automatic server a workaround exists. You can do the same as above (run Ansible in a loop) but use its dry run mode. Each time Ansible reports that changes are required, the Jenkins job (or cron job) can notify you that your provisioned configuration has been altered and servers are not in their desired state.

To make sure that your Ansible code is actually doing what you think it should be doing, the solutions mentioned by Dave Swersky apply. Both InSpec and Serverspec are tools that verify in a different way that your playbooks actually does what you mean. A great way to execute these kinds of tools in test environments (even docker containers) is to use kitchen.ci which handles all the glue between the various infra unit testing tools, and execution of your playbooks/modules/cookbooks.

Kitchen.ci was originally used to test Chef cookbooks, but plugins exist for Ansible and other CM tools as well.

Answer 4

Test Kitchen has a kitchen-ansible provisioner plugin for testing of Ansible code. It isn't as deep as the Chef integration but it does get the job done for most cases. There is also the more recent Molecule project which is a dedicated Ansible testing system.

Answer 5

You can track configuration/infrastructure discrepancies/drift using Outthentic, it's easy to create a test suite to "fix" desired state and re-run it every time you need to track undesired changes.

Answer 6

It is usual to measure the code coverage for your source code written in python, Java, and so on. On the other hand, we usually do not measure the coverage for an Ansible playbook. Kirby is the tool to support this.