What is alert fatigue?

Question

Recently, we hear the term "alert fatigue" a lot. I have a few questions regarding this:

• where did this term come from?
• was it invented by a company like PagerDuty?
• what is alert fatigue actually?

Answer 1

The term Alert Fatigue, also known as Alarm Fatigue has been around for decades. It affects many professions, including medical, technical, and construction industries. According to the Alert Fatigue by Other Names reference, the earliest literature on the topic came during the Israeli Arab conflict. The practice/understanding of the issue is ancient.

No, PagerDuty did not invent the term.

Alert Fatigue:

occurs when one is exposed to a large number of frequent alarms (alerts) and consequently becomes desensitized to them.

Aesop's Fable about the boy who cried wolf is a classic example of Alert Fatigue. In Technology and Medical fields it occurs when alarms/alerts occur so frequently (with or without reason) that we become desensitized to them. The outcome of this desensitization:

dramatically reduces our reaction to similar subsequent threats.

In other words we are become so used to the alert that we either ignore it or take our time to acknowledge it.

References

Alert fatigue by other names
Alarm_fatigue
Cry Wolf - When Experience Becomes Fateful

Answer 2

I have spend years developing a variety of monitoring tools which employ a variety of approaches to track data points ranging from filtered logs, collecting system stats (i.e.: querying data from /proc or running a command and parsing output), and/or putting concise “tagging” code inside application workflow code to track overall performance. With all of these different approaches, the process of alerting when conditions are met hasn't changed all that much. From this experience, I have learned some key principles to avoid “alert fatigue” which can be described as simply getting so many alert messages by your pager, email, or mobile notifications that they decide can be ignored (either because they were told to or they took it upon themselves) resulting in your staff ignoring potentially important events, slowly insane, and/or secretly wishing for your early demise.

That said, these are few key principles in creating triggered alerts which I would say are relevant now as they were 20 years ago. Of course there are gobs of other principles to creating good monitoring and alerting so this is by no means a complete list.

• All alerts should be “actionable”. That is, if an alert notification is sent, it means something should be done by the recipient. Non-actionable alerts is the principle cause of “alert fatigue” because there is a precedent to ignore alerts which can result in situations like "I don't know what this means, so it must be junk". In contrast, to say an alert is "actionable" this means anything from validating something manually that for whatever reason may not be obvious to automate such as manually running a process so a subjective assessment can be made, or it could mean remediation steps like restarting something. Don’t page people in the middle of the night for specious reasons where it's unclear whether there is a potential problem or not. This may seem obvious, but it's common to create alerts that do with without realizing it, so one good practice is to make sure you receive any new alerts you create so that you have actual first-hand experience of what the support engineers receive so you can tune or adapt them as necessary, all they while communicating what you are doing so support engineers realize that you're "getting the kinks out" and don't immediately get in the mindset that this new alert is garbage. Once your experience indicates that after a time of tuning the alert conditions so that the notifications are commensurate to a condition that requires action, then communicate that it is "live" and should be treated as such. (Having a way to identify "alerts in development" is also helpful.) On the other hand, if you or your team has infrastructure that triggers alerts where you wind up telling support staff to “just ignore it” without fixing it in any way, you have a critical process problem. Ideally, this alert should be suspended until it is "fixed". In summary, “alert noise” creates the “boy who cried ‘wolf’ problem” and you eventually will have people ignoring things that will inevitably be the notice of actual site degradation and you will get burned. It's not if but when.
• Notification “insistence" should be appropriate to the problem. For example, when you have a production impacting event, day or night, appropriate people who can fix the problem should be paged where the expectation is that the alert gets acknowledged in a short time before automatically getting escalated, etc.... On the other hand, If the issue is something that can wait until the next business day such as a disk slowly filling up or a not production issue, send an email or message to a Slack channel to be addressed later, probably the next business day. Set up some system of priority that has a specific SLA.
• Use monitoring dependencies whenever possible to avoid alert “storms”. A proper monitoring toolset should allow you to set up a dependency relationships so that alerting is suppressed if an overarching issue would cause other monitoring failures which would depend on it. A simple example would be if a host is unreachable, send just a network unreachable alert. Don’t send alerts about disk, ports, etc... all of which require network connectivity to ever work and would just give you "unknown" results anyway. That just muddies the waters when you need clarity so you can quickly resolve problems by looking at your alert list for a couple seconds, not scrolling through pages of alerts.

Start with these three tenets first if “alert fatigue” is an increasing issue in your team. If after doing this and you get to a point where your alerts are meaningful and concise but are are overwhelming in sheer numbers, you probably need to focus on creating automated recovery as the next phase of your monitoring endeavors starting with incidents where the run-book is discrete. For example, if a web service is unresponsive because of an exception that it cannot recover on its own, instead of sending an alert, attempt a restart. Send the alert only if the restart doesn't resolve it and send a non-urgent message if it does which can be noted during the next business hours.