How to monitor a REST call response?

Question

My project is using customer.io to send emails in different flows.

Every now and then we have a bug, and one of those calls stops and email is not sent.

I want to get alerted when an email is not sent in the last 24 hours. It would be great if the solution could give me alert on any spike but the mvp is 0 in the last 24 hours.

What is the best way to do it?

Currently, I am trying to use some process to poll customer.io REST API which provides exactly the information I need.

[{
    "campaign": "Hello", 
    "sent": 0 // This should trigger an alert. sent is for last 24 hours
}, { 
    "campaign": "World",
    "sent": 1 // should not trigger an alert
}]

Then to do some ETL on this, and feed some data source (maybe cloudwatch, haven't decided yet), and then hook grafana to give me alerts on any spikes.

I am new to this field and am wondering if I am going the right way and if there's a better/easier way to tackle the problem.

== Update - I found out about Grafana SimpleJSON plugin but it expects me to be the owner of the backend. I am going to write some lambdas and use gateway (basically implement an ETL) and use SimpleJSON plugin from grafana to get things going.

Answer 1

There's no generic answer to this, but a ton of options. Usually the right choice for you would be dependent on what monitoring and alerting stack you're already using.

For instance, if you are using Graphite, Prometheus, rrdtool, or any other round-robin database to record a metric when you're sending out emails, those all have APIs that allow you to query them across a time range to see what the values were. And depending on what alerting system you're using (Nagios, Icinga, Sensu, etc.) there is probably already a plugin that allows you to easily interact with that particular metrics database.

From another perspective, if you're logging when you send emails, then a log system like ELK will allow you to search and see if any of those log lines are found. Again, the various alerting systems will have plugins to most common log systems. (There are also individual systems designed purely for those tools, like ElastAlert with its flatline type for ELK.)

Thirdly, you could approach it the way that you are, by checking in with customer.io's API on a regular basis. But no, there is not one standard tool for doing this, but rather one (or at least one!) per alerting system.

Finally, you might consider doing synthetic checks instead of or in addition to your real-user monitoring. This means having an automated test that performs actions that you expect to result in sending an email, and then checking to see that an email gets sent.