I would like to monitor for recent spike in failed application jobs at my company.
Job statuses are stored in a MySQL table. We have on order of 50 different types of jobs, and sometimes, one of those jobs will start to fail, either because of a change in code or a change to a contract. In Production we'd like to visualize the spike in failures and alert on it.
We have ELK, grafana, opsgenie, and python, bash scripting capabilities. ELK doesn't seem to work because it parses logs, and our logs are not well standardized that we can detect a spike in failures. Grafana seems to work better for metrics like CPU and memory. We've done montioring in the past, by scripting, but surely there must be a better way. Customers are hit with the problem long before we know about it.
Is there something in ELK, Grafana, or another open source tool that can help us here?