Our code is written with:

  1. Unit tests
  2. End to end tests
  3. Code review
  4. Staging process
  5. Deployment process

On the contrary, our alerts are just written and then modified occasionally manually. No quality process at all.

This process is reasonable for simple threshold checks. However, our alerts are sometimes built on complicated queries. Sometimes composed of ~20 lines of a query.

If we accidentally break an alert, it could expose us to production instability since we won't know if some logic or component breaks.

The question

Is there a recommended methodology for validating the quality of complicated alerts?


We're using Splunk alerts



As monitoring is part of production it should be part of DTAP as well. This implies monitoring in Development, Testing, Acceptance and Production as well. If somebody modifies a check in production and it will silence some check and if the team does not get reported that there are issues then the customer will be impacted and that could be a huge problem. In short, if monitoring is part of production it should be applied to all stages of DTAP as well.


If custom scripts are used in splunk and these are written in Python or some other language, then you should also apply unit tests and integration tests. In most monitoring systems, a 0, 1, 2 and 3 exit code are checked so this should also be covered in the tests. If a monitoring is written in bash, one could use BATS and Powershell can be tested as well with Pester.

Why testing these scripts? Again, same reason as depicted in the DTAP section. Imagine that somebody breaks some monitoring script by introducing a typo and you do not get notified, then this could have a huge impact on the customer and also on the team. Imagine that you have to debug for a couple of days because some script is not working well, while that could have been prevented. So I advocate to apply unit tests, integration tests and even CI for this "simple" monitoring scripts.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.