11

Infrastructure as code tells us to use tools that automate your builds. Great. Tools like ansible, chef, puppet, salt stack and others push us towards writing how infrastructure looks like, while resolving the differences.

In Salt Stack those bits are called states. If state does not match the reality, the tool will resolve it for us. In other words - we are writing a test for our infrastructure and if the test fails, the tool will fix it on its own. At least that's the idea.

XP teaches us to use TDD and the question is if it's applicable to infrastructure? The tooling suggests it is.

I can imagine few type of tests that can be very useful.

We write smoke tests that are bundled with the deployed service to ensure that end-to-end the deployed service works and runs as expected. This would be an API call or/and systemctl check to make sure what we just deployed works. A lot of this functionality can be covered in the same states since tools like ansible have states to make sure a service is running.

There is project Molecule that allows running individual roles (as ansible calls its states) against docker or another temporary virtualisation engine. This forces to decouple roles and allows to execute them in isolation from the playbook while working on them. Tests mostly allow mocking the variables that the role is supposed to work with. Other examples seem like a duplication of the ansible engine though (assert a file belongs to a user...).

ThoughtWorks tech radar right now praises tools like inspec, serverspec or goss for validating that the server meets the spec. But we are writing a spec, aren't we?

So is there a point in further testing infrastructure if we are describing infrastructure in states/roles? I could suspect this becomes more required in larger organisations where one team provides the spec and other follows, or if there is a large set of roles maybe you want to run a subset of those and get a speed benefit from tests? I'm struggling to see why you would write a test if you could have a role/state for the same question in mind.

6

In short, I see two categories of tests for your infrastructure: 1) does it have everything you need to run your application and 2) does it not have any superfluous stuff.

First and foremost, you can treat the test suite of your actual software as a kind of "meta test" for your infrastructure. As long as you create the infrastructure from scratch for each test run, and the test suite runs completely on that infrastructure (i.e., doesn't use outside services) the fact that the whole suite is green means that your codified infrastructure is sufficient as well.

Second, especially from the security perspective, you can write tests against your infrastructure. I.e., if one part of your infrastructure is a VM running Linux, you might write a test that does a port scan against that VM, to make sure that there are no unintentional ports open, which may have been installed by an unintended apt-get install side-effect. Or you could write tests that checks whether any unexpected files were changed after your proper test suite has completed. Or you could check the ps outputs of your VMs or Docker containers for unexpected processes and such, build white-lists etc., and thus get automatic notification if some 3rd party package changed in an undocumented (or unnoticed way) in some upgrade.

These second type of tests are, in a way, similar to what you would do in a classical ops setting anyway, i.e., hardening your servers and checking for intrusions, avoiding full ressource and such.

  • So after some time, this is exactly the stance I took. Parts that are executed by ansible are not tested by itself, but the side-effects of those actions are tested using goss. So for e.g., RPM is installed (ansible) and then tested if the expected default file is put in place, or the service is running and listening for a specific port. I don't want to fix such issue automatically, but be notified and stop the progress. Sure Ansible can test the system for you as well, you just need to be explicit about it, but in our case, we use goss to test the behaviour of service within a container – JackLeo May 18 '18 at 11:32
1

IMHO it's rather redundant to write TDD tests for items entirely covered by the IaaC state specification. Doing so implies that the effectiveness of the IaaC is questionable - why would you use it if so?

Looking at it from a different prospective IaaC itself (if/when done properly) incorporates capabilities already tested and considered to be functioning reliably. Which is what makes it attractive and which makes writing TDD matching tests redundant.

For example, an IaaC configuration specifying a system with SSH being installed already incorporates reliable checking for SSH being correctly installed and, if not, mechanisms for properly installing it. Which makes a TDD test for checking if SSH is installed redundant. If your IaaC config also specifies sshd to be started and listening on a specific port then TDD tests for sshd running and listening to the respective port would also be redundant.

Note that my answer is not targeting TDD or any other type of testing that checks if your IaaC configuration as a whole fits a certain purpose. That remains valid and can be used in TDD, CI or similar checks during development of that IaaC specification - I believe @AnoE's answer is applicable in such case.

  • Do you apply same thinking to ensuring SSH (or whatever) is enabled on a specific custom port, which is parsed from an external configuration file? That’s not resting on tested units, that’s novel logic. – Jon Lauridsen Dec 4 '17 at 18:28
  • 1
    It should be part of IaaC, if it supports it. If not - then this discussion doesn't really apply. Yes, this could slide into how much stuff can be covered by IaaC, but that's a different discussion. – Dan Cornilescu Dec 4 '17 at 19:06
  • 1
    I'm also assuming we're not in a context where redundant checks are required - in some cases redundant checks following completely different code paths or even infrastructure may be required. – Dan Cornilescu Dec 4 '17 at 19:08
0

It looks like everybody here assumes an IAC tool always run as expected, but I can tell (from my own experience) this is not always the case, otherwise unit test would actually be useless.

I remember a picture saying "Ansible playbook ran, everything is fine" with a building burning in the background ...

Running a declarative state and having the server in this actual declared state are 2 different thing from my point of view and experience at least.

A broad and heterogeneous environment, spread across multiple DC, reachable through public network etc... There are multiple reason for which a state could not be applied, either fully or partially.

For all these reasons, there is room for unit test allowing one to get a snapshot of the actual server state, which, again, might differ from the aimed state.

So I'd say yes, unit test are useful even in a IAC managed environment.

EDIT

What about the non-regression side of the dev branch of the IaC code base ? so you'd make changes to your code in dev branch and merge it to the prod branch hoping for not breaking everything ? unit test are so valuable, and usually simple to implement I do not see why one would code without this feature.

Reference (in french sorry for that): https://fr.slideshare.net/logilab/testinfra-pyconfr-2017

  • 1
    That'd be at least polite to add some comment with a down vote, don't you think down voter ? Especially on this kind of question where the debate could be very informative or even constructive. – Pier Mar 22 '18 at 14:18
  • I assume the tone of your answer which is quite aggressive to everyone who has interacted with this question added to the fact you're not giving any reference out of your own exemple nor describing any observed malfunction was the reason for the downvoter. You're finally ending up saying the same thing as other answers says, have smoke tests in your provisioning system to ensure the system is OK, which most systems does by failing if they can't converge the system in desired state. Concerning your edit, usually the merge is done after deploying to staging and ensuring the smoke tests pass ... – Tensibai Mar 22 '18 at 15:12
  • I definitely did not intend to be agressive, I am not using my natibve language here (this is obvious I guess :) ). – Pier Mar 22 '18 at 15:26
  • We may discuss it in DevOps Chat if you wish, I think I saw this presentation or one like it at a devoxx event few years ago. I slightly disagree with calling that unit test, that's more functional tests. – Tensibai Mar 22 '18 at 16:00
  • Some one from my dev team told me the very same this was not unit test, I am no dev so I may be wrong about calling this unit test, definitely – Pier Mar 22 '18 at 17:16
0

In my experience one of the main differences between Dev and Ops are "heavy run time dependencies". Installing packages heavily depends on repositories, networks or valid keys, or lets say instantiating a new cloud server - it depends on your provider resources.

In terms of server-provisioning even if you did not change your provision code, your image will be valid most of the times but sometimes not. So I think testing is really essential for providing working images.

If you go beond single servers things get even worse ... how you will test reachability in whole network-setups? Including DNS-resolution, routing and firewalling? Even if your IaaC providers API works as expected (I've seen wired issues in this area) I really like TDD in this case.

Because I've not found any testing tools in this area, we wrote one in our spare-time: https://github.com/DomainDrivenArchitecture/dda-serverspec-crate

So I think TDD is really important in the DevOps world!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.