7

I can't imagine having only one person on-call at a given time. There are several reasons that the primary person could be unable to handle an issue - perhaps they are unable to receive alerts, they have limited Internet access for a period of time, they or a family member could be ill and they are unable to work yet the on-call rotation hasn't been updated ...


4

Yes! And consider a third and a forth person if you're large enough. Folks can get sick. Their phone might have run out of battery. They might be dealing with a major local power outage. Folks go on vacation and forget to update the pager schedule. All of that, and more, can be addressed in a more reliable way by having a timeout on ack'ing a page ...


3

Within the CloudWatch integration in PagerDuty, you can change the "Derive name from" field to something like "Alarm Description".


2

There are a number of things that I think it's important to know before going on-call: How to triage an issue. There should be a defined list of priorities with examples so that you can identify how serious a problem is. What the priority levels mean. Can you ack this and then go back to sleep? Who to contact post-triage. The on-caller's primary ...


2

The easiest way to build this would be to have some sort of metrics-based alerting system like Prometheus, datadog, etc. Those systems allow you to have a counter that gets incremented and you can see how many incidents happened in a given period of time on a pretty graph. Most metrics systems will tie nicely to PagerDuty for the alerting piece. Having ...


2

Based on the PagerDuty documentation on Cloudwatch, it doesn't look like it's configurable: https://www.pagerduty.com/docs/guides/aws-cloudwatch-integration-guide/


2

3-4 years experience with PD here. PagerDuty will not monitor itself in the way you want, it's made to push notifications, calls, alerts and etc. As mentioned in the comments you should use maintenance windows, disabling a service means you know what you are doing, that's not something I have ever done except for 1-2 testing purposes for the last 4 years. If ...


2

PD has a lot of native integrations that can do auto-resolve but for those that don't you can use the API and/or rulesets to do that. Here is a link to the ruleset documentation: https://support.pagerduty.com/docs/rulesets And here is documentation on the incident resolution API: https://developer.pagerduty.com/api-reference/reference/REST/openapiv3.json/...


1

We ended up creating a new PagerDuty service with a schedule attached to it. That schedule only had anybody assigned for half of the day. This ended up duplicating some info from the service we already had, but it was the least painful way we could find to deal with it.


1

Here's my two pieces of advice: Work out schedules so that you're not on-call 24 hours a day. There should be a hand-off between daytime operations and after-hours support. I encourage people to split half their regular time between support weeks and project weeks; after-hours support should come during a project week. Get in the habit of taking notes ...


Only top voted, non community-wiki answers of a minimum length are eligible