What is a good strategy to reliably and automatically perform remote system tests of a web service?

Question

In a company I work for, we have deployed many services: a major part of them are small and rather rarely used, but an outage would be a problem nonetheless. At some point I realized that manual testing of whether they work properly requires too many resources and it might be a good idea to automate the work.

My first thought was that self-hosting this piece of infrastructure means that I'd have to maintain two separate critical ecosystems: if I run the tests on the same server (or even hosting provider) that my services run on, downtime might bring both systems down and I would get no notification of a failure. Because of that, I decided to choose paid solutions.

I researched the topic and found many tools like Runscope, but most of them were rather basic request-response validation tools: if the server doesn't time out or respond with an error, the tools assume that everything is OK. This doesn't work well when a website is not open to unauthenticated users: crafting a POST login query is in many cases not feasible (e.g. because of CSRF) and means that I need browser-based automated tests, as opposed to query-based ones. I was suggested to expose an unauthenticated "sanity test" endpoint, but this feels too synthetic: ideally, the test would resemble user interaction as much as it's reasonably possible.

My current solution is to use Pingdom's transaction/synthetic tests, but I quickly discovered multiple problems:

1. Pricing. Pingdom charges 490,00 € yearly for 10 automated browsed-based test scenarios with failure notifications,
2. Security. The current solution requires me to upload credentials to the tested applications in order to test anything beyond the login wall,
3. "who's watching the watchmen" problem. I solved this by exposing a separate service which keeps track of when it was last contacted and sends a separate set of notifications if Pingdom didn't contact it for some time.

I feel like this is such a common problem, I cannot believe there are no popular/standard solutions or strategies that would address it. Is my approach somehow wrong, or is it something that is usually developed in a project-specific manner?

Answer 1

For less trafficked parts of your web service, synthetic transactions are a good option.

There are some trade-offs to be made when choosing either using an API to or using a full browser style check.

API checks require changing the application being tested. I'd try and go this route first, but it is dependant on the functionality you need to test. These can be considered health endpoints. Some example implementations are Beamly S4 or the Hootsuite Health Checks. There are many sites which can hit these endpoints as you mentioned eg. Pingdom, FreshPing or Datadog Synthetics.

Full browser style checks do not require changes to the application, which means they are easy to add. As they use contents of the rendered webpage to navigate around, they can be brittle. Changes to the layout or CSS of the page might break the tests that have been created. Options here include BrowserStack or Datadog Synthetics.

On running this yourself, you can also run into the problem that certain locations, regions, POPs for CDN's might have issues reaching your site. This is why most of these services offer many locations to hit your site from, so you can catch those issues which might only affect some of your customers. So I'd definitely recommend going the hosted route.

From a security perspective, these systems will definitely need credentials. If you can't lock down the account used by these services in your application to read only, or only provide permissions to certain areas on your app, an option might be to rotate credentials. For example, create job that runs daily that changes the service accounts, and updates the provider with the new password. For example, using the Datadog Synthetics Terraform resource.

For "watching the watcher": As you will have regular load on your application now based on the synthetic transactions, you could instrument your application or parse the logs to ensure the transactions are coming through.