What is the best way to monitor job status?

Question

I would like to monitor for recent spike in failed application jobs at my company.

Job statuses are stored in a MySQL table. We have on order of 50 different types of jobs, and sometimes, one of those jobs will start to fail, either because of a change in code or a change to a contract. In Production we'd like to visualize the spike in failures and alert on it.

We have ELK, grafana, opsgenie, and python, bash scripting capabilities. ELK doesn't seem to work because it parses logs, and our logs are not well standardized that we can detect a spike in failures. Grafana seems to work better for metrics like CPU and memory. We've done montioring in the past, by scripting, but surely there must be a better way. Customers are hit with the problem long before we know about it.

Is there something in ELK, Grafana, or another open source tool that can help us here?

Answer 1

There could be multiple ways to solve this.

1. Create a python based cronjob that queries the DB every T secs (scraping interval) for the failed jobs in the last T secs (delta). Push the relevant data into a specific index like failed-job in ES. Then use that index to set up alerts in Grafana/opsgenie.(You can also use watcher here) Don't need to read logs.

2. Use prometheus,(if already using), create a new custom prometheus exporter.Exporter would be a Flask web server that provides an endpoint to run the query against the db and return results in a prometheus compatible format to be stored. Configure job in prometheus to scrape this endpoint every T secs. Again, use Grafana for visualization.

3. Use a component called , prometheus-pushgateway, if possible, either push the job status into this component using the python script as in option 1. or modify the Job itself to push the data directly. Again , can be connected to Prometheus/Grafana.

If you are trying to solve only a single use case as described here, option 1 would be the better choice. For a larger more complex setups that covers a lot of other use cases, Prometheus might be worth looking into.

Alternatively, you can use the same design to push metrics into any datasource.(Like the one you are using for storing CPU/mem metrics, as specified.)

Answer 2

Not sure if it checks all your boxes but Rundeck is great for having an interface for your jobs . It alerts you if a job fails , has a quick summary for the jobs. here you can a look for details.

Answer 3

Maybe you can connect Grafana to the MySQL database and query for number of failures? Grafana also has a basic alerting capabilities, so you could send alerts to the interested parties.

At my company, we have a similar approach to jobs - the statuses are kept in the database. What we do to check for failures is we use Nagios and a custom query to check for problems and send alerts - but you might want to stick with Grafana and not install another tool.