3

We have a small team, and we're just getting started with PagerDuty. We have a very noisy service that has a lot of testing goign on around it. At times, we've had to turn the service off to keep pagers from blowing up, but then forget to turn it back on.

Is there any way to get notified when a service has been disabled?

3
  • I believe PagerDuty has built-in option to suspend its notifications for X amount of time. Would it solve your problem?
    – taleodor
    Aug 17 at 20:00
  • Are you referring to maintenance windows? Not quite; but we could use it to disable while we're testing. I was more thinking of something along the lines of a notification that gets triggered when you pause the service itself. Aug 17 at 20:20
  • 1
    Yes, was thinking of maintenance windows; not aware of other solution.
    – taleodor
    Aug 17 at 20:39
2

3-4 years experience with PD here.

PagerDuty will not monitor itself in the way you want, it's made to push notifications, calls, alerts and etc. As mentioned in the comments you should use maintenance windows, disabling a service means you know what you are doing, that's not something I have ever done except for 1-2 testing purposes for the last 4 years. If you are forgetting to enable the service.. then there is another problem that is not technical related, you should not be forgetting that step! I would suggest you don't disable the service, use maintenance windows, if the service is noisy, why put it in PagerDuty, monitor it when it's fully down, or use some other rule to do this. What I would do:

  1. Evaluate what is a critical alert.
  2. Should you be notified via pager duty for every single thing?
  3. Probably set more dynamic alerts, that the alert will be triggered only in specific hours(work hours). Link to PD documentation
  4. Try not to disable the Service, use maintenance windows.
  5. Decide what kind of escalation policies do you need/want.

EDIT: Never tested this, but PagerDuty has an API that possibly could be used to get a status of a service, but something has to listen/run that API GET request and send it to another PagerDuty Service, that can be used to monitor what you wanted, kinda. API Service Documentation

EDIT 2: I tested the API, if you use that method, you can use something else to monitor for xx amount of time if the service is disabled, by getting this data out of the request: "status": "disabled". For example any Monitor that can send a GET request and get this data, can be set to "If status: disabled for 1 hour, send PagerDuty alert". And you can have a separate PagerDuty Service for example that listens for this monitoring.

enter image description here

3
  • Thanks for this; it had fallen off my to-do pile. "If you are forgetting to enable the service.. then there is another problem that is not technical related, you should not be forgetting that step! " - yep. We know. But just because people SHOULD do something, doesn't mean it always gets done. If something is critical, I'd like to be able to monitor it. Aug 25 at 10:36
  • I understand completely, thats why there are specific rules that if ignored too often lead to.. not so fun actions. This is one way to monitor it if needed, im interested if you come across another way to do it, to share it, as that will be a very helpful insight. Aug 25 at 11:18
  • 1
    The API is an awesome idea; we were already using it to publish our on-call calendars to folks that don't have access to PD. We can build on that to report on which services are running or not (as well as send it to PD for notice when other service are down). Aug 25 at 14:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.