3

Initial Problem

We have some badly behaved overnight maintenance jobs that cause alerts for AppDex and error rates for our apps. These are metrics that would be problematic if they happened during the day when customers are using the service, but at night there are absolutely no customers so there's no motivation by devs to make the nightly jobs less error-prone. So now I want to make it so it doesn't alert every night. Between New Relic and PagerDuty I would think I could figure that out, but so far I've come up with nothing practical.

New Relic

I haven't spent tons of time with New Relic so I was learning as I tried adjusting things, but based on this thread on New Relic's site it seems clear that time-boxing alerts is not a feature they current support or will in the forseeable future. I've dealt with using history as a predictor for future results using a few technologies and it is a mess that would not be a reasonable replacement for the simple alerts that are in place currently. Going down that path would create more noisy alerts without detecting any more problems than we already find.

[ It doesn't seem very DevOps-friendly to me to ignore feature requests for simple features that would solve lots of people's problems and only offer complicated quagmires as an alternative. I'm sure they're all jazzed about 24-7 cloud stuff, but ignoring the market of people who have prehistoric needs is sad. My audience is literally captive and they are going to stay that way until we stop putting people in prison. This use case will not go away regardless of how much cloud pixie dust you sprinkle on it. ]

PagerDuty

I've used PagerDuty merrily for years and I was sure there would be some way to handle this after the alert left New Relic, but:

  • Schedules apply to people and come after an alert has been through a Service. If you could add a Service to a Schedule we could hack this, but no, that's not there.
  • Event Rule-based Suppression doesn't include time-of-day conditions.
  • Maintenance Windows might work, but I was really hoping this wouldn't turn into another cron job I have to keep an eye on.

The Question

Is there any way to get New Relic or PagerDuty to not alert during nightly maintenance?

Improve this question
1
  • How did you end up solving the problem? Maintenance windows?
    – Kaushik Thirthappa
    Commented Sep 2, 2019 at 10:50

1 Answer 1

Reset to default
2

We ended up creating a new PagerDuty service with a schedule attached to it. That schedule only had anybody assigned for half of the day. This ended up duplicating some info from the service we already had, but it was the least painful way we could find to deal with it.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.