I'm looking for a platform that allows me to run tests against multiple, closely-coupled systems. We deploy different products to Red Hat family machines - think Rocky Linux, CentOS, Fedora, RHEL, etc. This is done using SaltStack, which allows us to completely set up blank VMs (in development they are built on QEMU using Vagrant, on a linux host). To date, regression testing has been done by hand, and the extent of our automation has been Test Kitchen ensuring that highstates return clean, coupled with some basic InSpec profiles in some cases (is port listening, etc).
It has become necessary to perform more testing as these systems become more complicated. For example, take a cluster of three VMs, which each have services used by the others, all heavily configured with SaltStack. It seems that the standard advice for testing is to test each system individually with tools like Chef's InSpec, or testinfra. However, I'm looking for a way to test the cluster overall.
I'm looking for a way to test all my services in different cases. For example, I should be able to kick off a test that:
- Highstates one blank machine
- Uses a client VM to access the public service of the first (i.e. a basic MySQL query)
- Highstates a second server VM
- Tests both services again
- Disables a network interface on one VM to test split-brain functionality
I could do all of this with a long bash or Python script and many SSH commands to each VM, but I would prefer to not reinvent the wheel. Is there a testing framework that can run tests on multiple machines, in order as I define, that allows me to test the whole system in aggregate?