monitoring/alerting system which can support very arbitrary data source and logic

Question

I am looking for a monitoring/alerting tool/system which can support very arbitrary data source and logic.

(following is a made up example to more easily demonstrate my needs)

I have a database where I can find out the query execution times for my clients(which are individual companies). The query times are expected to vary greatly between clients, because of the data of each company. I want to have an alert when the query time for a client is ?% over some median value from the past, and where I am comparing Monday's data against previous Mondays' data(I am being difficult for demonstration purpose).

I want to avoid defining a separate alert for each company client.

I want to avoid defining a separate alert for each kind of query I want to monitor.

I want to avoid having each source pre-push data to one central place.

I would like to monitor some other things that has nothing to do with databases, and want to apply some entirely different logic for my alerts - e.g. some REST api call to some other system, or some ELK or time series data, or some hardware monitoring data.

I am contemplating a custom solution, but want to see if there is already a solution out there that can meet some or more of my needs. I have seen/used some monitoring solutions out there such as Nagios, though not an expert on those. Will appreciate any pointers or recommendation.

Answer 1

Consider using something like nagios or Zabbix. While it might be nice to avoid custom queries and alerts, this is impossible. The odds of someone using your specific suite of tools in your configuration are slim-to-none - and the odds of having pre-canned monitoring for that suite in that specific configuration are even slimmer. Simply put, there is no way to avoid this custom monitoring. While monitoring, alerting and reporting management and configuration can be tedious, there is no easy shortcut for doing it properly and it's probably going to take more time, dedication and resources to do it properly than you think.

Out of the 4 monitoring systems I have used extensively, I must recommend nagios - it seems to be the best suited for configuration management systems and most compatable with DevOps philosophy.

It may sometimes be useful to use a system such as elastalert or similar systems as middleware for alerting/monitoring/reporting, but you will have to judge each situation/monitor on a case-by-case basis, but you will find these tools (and most monitoring systems) have some SDK, API or tool capable of helping you with your monitoring customization. Just choose the easiest to configure system with the lowest ROI and support cost and be sure to stick with it and follow your monitoring migration project to completion. Be sure to consider dedicated resources/personnel for this task if possible.

Answer 2

Adding my brick to the list here, I'm at a point where updating continuously the monitoring system to add new instances and cleanup old ones is a pain.

As such I've turn toward prometeus which works with a simple contract, an exporter does the collection job (could be the cloudwatch_exporter on AWS or when using netdata querying it (either a central one or each separately, but the latter lost the advantage of auto-discovery) (doc for the netdata part here) and there's also a node exporter from prometeus team.

The main advantage behind prometheus is the use of mathematics functions and grouping in the alert definition by label from the metric name. This allow to define alerts on various way, for my infrastructure monitoring I work with standard deviation or standard variance to avoid fixed level of alert on cpu usage, it alerts me if the stddev goes up by 20%.

So for your case you could export your database or push the metrics to prometeus and assuming a metric name client_queries with the client name as the label client you could do something along the line of this (untested of course and just to illustrate):

ALERT ClientQueryDriftUp
  IF avg(client_queries[5m]) > avg(client_queries[5m] offset 1w) * 1.2
  ANNOTATIONS {
    summary = "{{ $labels.client }} query has gone up over 20% of last week",
    description = "{{ $labels.client}} query average is high ! (current average value: {{ $value }}s)",
  }

Or if you just want an alert if the difference between averages is over 1 second (assuming values are in milliseconds) you can use this IF condition:

abs(avg(client_queries[5m]) - avg(client_queries[5m] offset 1w)) > 1000

Of course this answer is just an overview on how it can tackle your problem and is far from exhaustive about it.

Answer 3

I would suggest elastalert as an alert system for elasticsearch, it has amazing capabilities, you can execute any query on elasticsearch and define an alarm depending on the response and it support multi ways for alert like sending email or slack or telegram, even execute specific commands.

For monitor data in elasticsearch as you are already using ELK, you can use Kibana for monitoring not only for making elasticsearch query and create very valuable Dashboards

For hardware and services I think Nagios do very good job there, but for more easy solution you can use Monit it's very easy to use and configure and can do a lot of jobs and service management, But the same as you are already using ELK, you can configure metricbeat to monitor the server (CPU, RAM, Disk usage, etc...)

For me I think the best thing is to use ELK with all the integration that you need, that will allow you to more focus on, and centralize your monitoring and alerting systems

Answer 4

We use Datadog's anomaly detection for this purpose:

https://www.datadoghq.com/blog/introducing-anomaly-detection-datadog/

We push custom metrics which in your case would be something like query execution time tagged by client name, and you can create a single Multi monitor that would alert based on anomalous behavior on this metric.