2

My project is using customer.io to send emails in different flows.

Every now and then we have a bug, and one of those calls stops and email is not sent.

I want to get alerted when an email is not sent in the last 24 hours. It would be great if the solution could give me alert on any spike but the mvp is 0 in the last 24 hours.

What is the best way to do it?

Currently, I am trying to use some process to poll customer.io REST API which provides exactly the information I need.

[{
    "campaign": "Hello", 
    "sent": 0 // This should trigger an alert. sent is for last 24 hours
}, { 
    "campaign": "World",
    "sent": 1 // should not trigger an alert
}]

Then to do some ETL on this, and feed some data source (maybe cloudwatch, haven't decided yet), and then hook grafana to give me alerts on any spikes.

I am new to this field and am wondering if I am going the right way and if there's a better/easier way to tackle the problem.

== Update - I found out about Grafana SimpleJSON plugin but it expects me to be the owner of the backend. I am going to write some lambdas and use gateway (basically implement an ETL) and use SimpleJSON plugin from grafana to get things going.

  • 1
    I assume you could use jq to parse the output and count the failures, but IMHO the application sending the mail should handle the failure itself, retry, and alert after giving up trying a few times. – Tensibai Nov 7 '18 at 21:00
  • @Tensibai - there's no failure. human error caused some code to stop triggering. there's no logs no nothing.. I want this to be reflected somewhere. – guy mograbi Nov 7 '18 at 21:08
  • Then request the api to get this Json, then parse it with your language of choice, a 'cheap' solution is curl to get the Json then jq to count the failures or filter the failed events depending on what you wish as output I think – Tensibai Nov 7 '18 at 21:31
  • Just saw your update, this probably should go into a self answer instead of an edit – Tensibai Nov 7 '18 at 21:32
  • @Tensibai - not sure. Is this the easiest/recommended way to monitor REST responses? I prefer to avoid code in general when monitoring. Since we are using standards, I expect there to be a tool. Otherwise what's the point? – guy mograbi Nov 7 '18 at 23:43
1

There's no generic answer to this, but a ton of options. Usually the right choice for you would be dependent on what monitoring and alerting stack you're already using.

For instance, if you are using Graphite, Prometheus, rrdtool, or any other round-robin database to record a metric when you're sending out emails, those all have APIs that allow you to query them across a time range to see what the values were. And depending on what alerting system you're using (Nagios, Icinga, Sensu, etc.) there is probably already a plugin that allows you to easily interact with that particular metrics database.

From another perspective, if you're logging when you send emails, then a log system like ELK will allow you to search and see if any of those log lines are found. Again, the various alerting systems will have plugins to most common log systems. (There are also individual systems designed purely for those tools, like ElastAlert with its flatline type for ELK.)

Thirdly, you could approach it the way that you are, by checking in with customer.io's API on a regular basis. But no, there is not one standard tool for doing this, but rather one (or at least one!) per alerting system.

Finally, you might consider doing synthetic checks instead of or in addition to your real-user monitoring. This means having an automated test that performs actions that you expect to result in sending an email, and then checking to see that an email gets sent.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.