Looking at trying to build some resilience into our Ansible setup which deals with provisioning and configuration.

I understand a few methods of testing on the configuration side of things but I'm wondering how best to implement testing on the provisioning side of things, and if there are any tools which can help with this type of implementation.

Currently a lot of our testing is done serially during the playbook which makes a lot of sense for stuff like "has service come up; is the vip available; has this async task finished" but what really concerns me is our ability to manage drift of configuration at both the application and provisioning layer (such as VM configuration). I'm aware Ansible isn't the best tool for working with configuration drift but I'm curious to see your own opinions.

If you have something to fully automate the process even better. (we have a few ugly scripts which report back in slack daily).

Note: Right now we have a few conditions where a reprovision might occur (e.g. rebuild from backup, critical systems issue) but typically it just loops through some of the ansible configuring tasks and thinks no more of it.

  • 2
    serverspec.org – Matt Schuchard Feb 28 '17 at 20:24
  • Does Ansible run only once upon provisioning? If not, what frequency is it being run? Just trying to understand the problem before offering a solution. – Woodland Hunter Feb 28 '17 at 22:04
  • 1
    Hi @Naphta any of the answers has solved your question please consider accepting it by clicking the check-mark. This indicates to the wider community that you've found a solution and gives some reputation to both the answerer and yourself. There is no obligation to do this. – Richard Slater Apr 15 '17 at 20:23
  • I'm aware Ansible isn't the best tool for working with configuration drift Please explain. – 030 Apr 26 '18 at 22:31

Some options out there..

Testing tools: Sorted by github stars

  • Serverspec - Ruby, most popular tool out there, built on ruby's rspec
  • Goss - YAML, simple, <10MB self-contained binary, extremely fast, can generate tests from system state
  • Inspec - Ruby, think of it as an improved serverspec, almost same syntax, made by the chef guys. Built to be easier to extend than serverspec
  • Testinfra - Python, has the cool feature of being able to use Ansible's inventory/vars

Major differences between them:

Ultimately, I would suggest spending a day experimenting with all of them to get a feel for them before deciding for yourself.

  • With the exception of Goss, all the frameworks can run against a remote machine (ex. over ssh). Goss only runs locally or in docker w/ dgoss.
  • All frameworks can be run locally on a server, but require Python or Ruby to be installed or embedded. Inspec provides a self-contained <150MB bundle with an embedded version of ruby. Goss is a single <10MB binary with no external dependencies.
  • Goss has built in support for nagios/sensu output, this allows for easier integration with monitoring tools.
  • Goss tests tend to be simpler, but less flexible since it's based on YAML. Other frameworks allow you to leverage the full power of the underlying language Python/Ruby to write tests or extend the tool's functionality. (simplicity vs flexibility)
  • Goss allows you to generate tests from current system state
  • Testinfra to my knowledge is the only one that has built-in support for ansible inventory and variables
  • Inspec is backed by Chef

Continuous/divergence testing:

  • Chef Compliance - works with inspec to continuously test your servers, paid product
  • Goss - Can be easily hooked into Nagios or Sensu. Also, supports exposing server tests as an http endpoint.

Testing harnesses for development:

  • kitchen - Testing harness tool, launches instance, runs config management code, runs test suite. Made by the chef guys
  • Molecule - Similar to test kitchen, but written specifically for ansible

Full Disclosure: I'm the author of goss

UPDATE: InSpec 4.x or above uses a mixed commercial / open source license - see comments.

  • 4
    I understand you're a little biased, but inspec doesn't need Compliance to run periodically. And there's no dependencies to manage, all necessary libs and framework (ruby) is bundled in the package which can be installed locally on each node, when running through ssh/winrm, inspec put itself in place. You point about exclusively external command which is not true, a lot of tests are done by internal ruby code. (I think it worth being noted) – Tensibai Mar 29 '17 at 14:00
  • I've updated the answer to correct the external command and bundled ruby for inspec. Can you clarify or send me a link explaining how inspec can be used on it's own to detect/report on drift? – Ahmed Elsabbahy Mar 29 '17 at 15:10
  • Well, the reporting part will be at your hand, it's indeed Compliance goal to make a representation. But you can run inspec in a crontab and just rely on the crontab mail when there's a test failing to alert you (for example). – Tensibai Mar 29 '17 at 15:20
  • All in all, tahnks for the edit, that sounds a fair exposition of your tool and others. I'm afraid this may become outdated quickly, but +1 anyway for the listing and pointers. – Tensibai Mar 29 '17 at 15:22
  • Ah, yeah.. all tools can be leveraged in that way. I was trying to provide tools (or integrations with tools) that provide better reporting. To my knowledge there's compliance, or wrapping the tools in a way that makes them nagios/sensu/some-other-monitoring-tool friendly. Ex: slideshare.net/m_richardson/… Apologies if it came off as biased initially, wasn't intended to be. – Ahmed Elsabbahy Mar 29 '17 at 15:24

The two tools I've seen for this are InSpec and ServerSpec. Serverspec is a Ruby-based tool that builds on RSpec. InSpec is inspired by RSpec and ServerSpec.

I've used ServerSpec. It's cool, but maybe not 100% stable. I've had problems with testing for specific versions of software on Ubuntu.

I've read the InSpec docs but haven't dug in deep. It does essentially the same thing as Serverspec.

Judging by the Github commits, it looks like work on ServerSpec has tailed off somewhat, whereas InSpec is just getting ramped up.

UPDATE: InSpec 4.x or above uses a mixed commercial / open source license - see comments.

  • 1
    "I've had problems with testing for specific versions of software on Ubuntu." I fixed this after noticing a question about it on SO: stackoverflow.com/questions/42417864/… – Matt Schuchard Feb 28 '17 at 23:13
  • To slightly clarify, InSpec emulates RSpec but does not build on it. – coderanger Mar 1 '17 at 0:49
  • 1
    @MattSchuchard Thanks for that reference! – Dave Swersky Mar 1 '17 at 3:16
  • well "not 100% stable" for a testing tool does not sound any good – ᴳᵁᴵᴰᴼ Mar 8 '17 at 1:10
  • InSpec is very good as a test tool, and makes it easy to install nothing on the server, whereas ServerSpec can only do this with some extra work. Some work would be needed to have InSpec use an Ansible inventory though - probably easier if inventory is in YAML format (Ansible 2.4+). – RichVel May 6 '18 at 5:54

When using configuration management tools, such as Ansible, the tool itself would be responsible preventing configuration drift. Once you used Ansible to set a certain configuration, repetitive execution of Ansible will ensure your configuration is as you defined it to be. This also requires your Ansible code to be written in a manner which is idempotent.

Given the above, testing provisioning can be achieved by executing your Ansible playbooks in a loop from some server. For example, a cron job, or Jenkins, can execute the playbooks every 30 minutes and report back on any failure. Having no failures means that your configuration is in check, having failures means that there is a problem getting servers into your desired state.

In a case where you cannot trust your code to be written as idempotent, and thus you cannot really execute Ansible repetitively in a loop from an automatic server a workaround exists. You can do the same as above (run Ansible in a loop) but use its dry run mode. Each time Ansible reports that changes are required, the Jenkins job (or cron job) can notify you that your provisioned configuration has been altered and servers are not in their desired state.

To make sure that your Ansible code is actually doing what you think it should be doing, the solutions mentioned by Dave Swersky apply. Both InSpec and Serverspec are tools that verify in a different way that your playbooks actually does what you mean. A great way to execute these kinds of tools in test environments (even docker containers) is to use kitchen.ci which handles all the glue between the various infra unit testing tools, and execution of your playbooks/modules/cookbooks.

Kitchen.ci was originally used to test Chef cookbooks, but plugins exist for Ansible and other CM tools as well.


Test Kitchen has a kitchen-ansible provisioner plugin for testing of Ansible code. It isn't as deep as the Chef integration but it does get the job done for most cases. There is also the more recent Molecule project which is a dedicated Ansible testing system.


You can track configuration/infrastructure discrepancies/drift using Outthentic, it's easy to create a test suite to "fix" desired state and re-run it every time you need to track undesired changes.

  • 1
    Welcome to DevOps.se Alexey. While your tool may help the user in this case, for this to be an answer here there should be a little more to convey how this is relevant to the question. Otherwise it looks like a link-only advertisement. – chicks Oct 4 '18 at 18:12
  • Hi! Sure, just given more details. – Alexey Melezhik Oct 4 '18 at 19:57
  • From my point of view, this does nothing more than what rundeck can do and is out of scope for this question. Specially as the answer doesn't explain how it can solve auditing a thousand servers and be alerted of a drift or present a human readable report for an auditor by giving some parseable output. From what I just read it does no more than inspec can do (for exemple) and give less usable output at scale while requiring a lot of external tools to do the job and thus being less portable. – Tensibai Oct 4 '18 at 20:21
  • "how it can solve auditing a thousand servers" - I could find anything about thousands of servers in the question – Alexey Melezhik Oct 4 '18 at 20:28
  • "and be alerted of a drift or present a human readable report for an auditor by giving some parseable output." neither I found this in the question. The obvious alerting of drifting is the fact that tests begin to fail ... – Alexey Melezhik Oct 4 '18 at 20:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.