How does one perform systems testing against multiple interdependent machines?

Question

I'm looking for a platform that allows me to run tests against multiple, closely-coupled systems. We deploy different products to Red Hat family machines - think Rocky Linux, CentOS, Fedora, RHEL, etc. This is done using SaltStack, which allows us to completely set up blank VMs (in development they are built on QEMU using Vagrant, on a linux host). To date, regression testing has been done by hand, and the extent of our automation has been Test Kitchen ensuring that highstates return clean, coupled with some basic InSpec profiles in some cases (is port listening, etc).

It has become necessary to perform more testing as these systems become more complicated. For example, take a cluster of three VMs, which each have services used by the others, all heavily configured with SaltStack. It seems that the standard advice for testing is to test each system individually with tools like Chef's InSpec, or testinfra. However, I'm looking for a way to test the cluster overall.

I'm looking for a way to test all my services in different cases. For example, I should be able to kick off a test that:

1. Highstates one blank machine
2. Uses a client VM to access the public service of the first (i.e. a basic MySQL query)
3. Highstates a second server VM
4. Tests both services again
5. Disables a network interface on one VM to test split-brain functionality

... etc

I could do all of this with a long bash or Python script and many SSH commands to each VM, but I would prefer to not reinvent the wheel. Is there a testing framework that can run tests on multiple machines, in order as I define, that allows me to test the whole system in aggregate?

Answer 1

Testinfra lets you use the pytest test framework to interact with and run tests against remote hosts.

A simple test suite might look like:

def test_mariadb_query(host):
    '''Connect to a remote host, run a SQL query using the `mariadb`
    CLI, record the result, and verify it matches our expectations.
    '''

    expected = "count\n15\n"
    res = host.run(
        '''mariadb example -e "select count from widgets where name='doodad'"'''
    )
    assert res.rc == 0
    assert res.stdout == expected

The above would run the test_mariadb_query test on any hosts specified on the command line, e.g:

pytest --hosts=ssh://root@node1

You can specify a list of hosts in the test file instead of on the command line:

import pytest
import testinfra

testinfra_hosts = ["ssh://root@node1"]


def test_mariadb_query(host):
    expected = "count\n15\n"
    res = host.run(
        '''mariadb example -e "select count from widgets where name='doodad'"'''
    )
    assert res.rc == 0
    assert res.stdout == expected

With this, you can just run pytest and it will do what you expect.

If you have tests that interact with specific hosts, you can create pytest fixtures for those hosts:

import pytest
import testinfra


@pytest.fixture
def node1():
    return testinfra.get_host("ssh://root@node1")


def test_kernel_version(host):
    res = host.run("uname -r")
    assert res.stdout.startswith("5.17")


def test_mariadb_query(node1):
    expected = "count\n15\n"
    res = node1.run(
        '''mariadb example -e "select count from widgets where name='doodad'"'''
    )
    assert res.rc == 0
    assert res.stdout == expected

The above will run the test_kernel_version test against all hosts specified on the command line, but test_mariadb_query will run only against host node1, regardless of the command line.

The downside to this solution is that it's built on top of a unit testing framework; since tests are supposed to be independent, there's not a built-in mechanism to run them in a specific order. However, there are plugins that attempt to address this issue.

If your ordering requirements can be written as setup/teardown tasks, pytest fixtures may be a viable solution.

---

You can use Ansible itself to run your tests, and this gives you complete control over the order of execution of things, but you will generally "fail fast" rather than "fail last" (execution stops with the first failure, rather than continuing and showing you all test failures).