In a company I work for, we have deployed many services: a major part of them are small and rather rarely used, but an outage would be a problem nonetheless. At some point I realized that manual testing of whether they work properly requires too many resources and it might be a good idea to automate the work.
My first thought was that self-hosting this piece of infrastructure means that I'd have to maintain two separate critical ecosystems: if I run the tests on the same server (or even hosting provider) that my services run on, downtime might bring both systems down and I would get no notification of a failure. Because of that, I decided to choose paid solutions.
I researched the topic and found many tools like Runscope, but most of them were rather basic request-response validation tools: if the server doesn't time out or respond with an error, the tools assume that everything is OK. This doesn't work well when a website is not open to unauthenticated users: crafting a POST login query is in many cases not feasible (e.g. because of CSRF) and means that I need browser-based automated tests, as opposed to query-based ones. I was suggested to expose an unauthenticated "sanity test" endpoint, but this feels too synthetic: ideally, the test would resemble user interaction as much as it's reasonably possible.
My current solution is to use Pingdom's transaction/synthetic tests, but I quickly discovered multiple problems:
- Pricing. Pingdom charges 490,00 € yearly for 10 automated browsed-based test scenarios with failure notifications,
- Security. The current solution requires me to upload credentials to the tested applications in order to test anything beyond the login wall,
- "who's watching the watchmen" problem. I solved this by exposing a separate service which keeps track of when it was last contacted and sends a separate set of notifications if Pingdom didn't contact it for some time.
I feel like this is such a common problem, I cannot believe there are no popular/standard solutions or strategies that would address it. Is my approach somehow wrong, or is it something that is usually developed in a project-specific manner?