Is it a good idea to have a secondary person on call for production operations?

Question

The primary person on call could need help in situations such as:

• There are too many issues goes on
• Primary needs to step out for a short amount of time
• There are certain activities that are better taken up by a secondary person so that the primary can focus on more critical issues

So, is it a better practice to have a secondary on call for production operations? If so, does it make sense for alerts to automatically go to the secondary person in case the primary person doesn't acknowledge them on time?

Answer 1

I can't imagine having only one person on-call at a given time. There are several reasons that the primary person could be unable to handle an issue - perhaps they are unable to receive alerts, they have limited Internet access for a period of time, they or a family member could be ill and they are unable to work yet the on-call rotation hasn't been updated yet. If you have agreements in which you need to start and/or resolve issues in a specified amount of time, having only one person to get notified and start the work seems very risky.

Having a clearly-defined second person is also useful if the first person gets stuck. There's someone who knows that they are on-call and is hopefully doing the best they can to remain in communication and have access to the resources they need. It makes it easy to know who that person is and get help when it's needed outside.

However, if you are having "too many issues going on" or have so many on-call issues that the primary needs to "focus on more critical issues", that would seem to be indicative of poor product or process quality. Situations that require someone being called off-hours should be rare and not the norm, and simultaneous issues should be very rare.

Of course, this is likely to vary by system. Companies that operate at scale of Amazon and Google are vastly different than a B2B service with half a million total users globally who use the system during working hours. Larger, more complex systems with a need for greater up-time are going to have more people on call and shorter thresholds before proceeding down a call tree than smaller systems.

Answer 2

Yes!

And consider a third and a forth person if you're large enough. Folks can get sick. Their phone might have run out of battery. They might be dealing with a major local power outage. Folks go on vacation and forget to update the pager schedule. All of that, and more, can be addressed in a more reliable way by having a timeout on ack'ing a page before escalating to the next level. Typically 30 minutes is reasonable to give somebody a chance to ack a page. After that it is wise to try to see if someone else is actually available.

Escalation Tiers

I've typically seen these escalations arranged into tiers. The earlier tiers tend to have a few folks operating in rotation while later tiers don't get much noise and often are the same person all of the time. While there are many variations on this theme I typically see something like:

1. junior SRE/DevOps engineer
2. senior SRE/DevOps engineer
3. manager of SRE or DevOps team
4. director of engineering or operations

I usually see 3 layers in practice. It is less common to see the Director there. The first two layers can be arranged to help people sleep more:

1. SRE where it is day-time
2. SRE where is is night-time
3. a manager

But even if you only have:

1. yourself
2. the other guy

that's much better than it just being one of you.