In https://landing.google.com/sre/workbook/chapters/alerting-on-slos/#pros_and_cons_of_alerting_based_on_burn_r I don't understand how the reset time can be 58 min when the alerting window size is one hour.
How can the reset time be shorter than the alerting window size?
Consider a service using the alerting rule from the book:
job:slo_errors_per_request:ratio_rate1h{job="myjob"} > 36 * 0.001
The burn rate is the rate at which we are exhausting our error budget. A burn rate of 1 means that we'll have exactly exhausted our error budget at the end of our reporting period. A burn rate of 2 means that we'll have exhausted our budget half way through our reporting period.
For a 99.9% SLO, an error rate of 0.1% constitutes a burn rate of 1. An error rate of 1% means a burn rate of 10, and a 100% error rate means a burn rate of 1000
If we want to alert when we've exhausted 5% of our 30 day error budget in 1 hour, then that means we are burning error budget at a rate of 36, which translates to an error rate of 3.6% for a 99.9% SLO.
Imagine a service with a constant rate of requests, that starts serving 100% errors at t=0 minutes
Here our SLI counts the total number of "events".
SLIs written in terms of some kind of discrete events, where an event can either be good or bad, have many nice properties: They range from 0-100%, where 0 is "nothing is working" and 100% is "nothing is broken"; I find this quite intuitive. By making all of your SLIs behave the same, you simplify tooling, alerting rules, and understanding.
With a request-based system, events are usually individual requests. So 0 errors, 1 request in minute at t=-2. I used 1 request per minute for simplicity. The result is the same if you have 1 request per minute or 1 million; it's the ratio of errors to total requests that matters.
t=-2, errors=0, total=1
t=-1, errors=0, total=1
t=0, errors=1, total=1
t=1, errors=1, total=1
...
If our monitoring system evaluates its rules every minute, we get:
at t=-1, our 1h average error rate is 0
at t=0, the 1h error rate is 1/60 = 1.666%
at t=1, the 1h error rate is 2/60 = 3.333%
at t=2, the 1h error rate is 3/60 = 5%
Our alert fires 2 minutes after the event started.
Let's assume our incident continues until t=60.
at t=60, our 1h error rate is 100%: a burn rate of 1000, for a 99.9% SLO
at t=61, the 1h error rate drops to 59/60 = 98.333%
at t=62, the 1h error rate is 58/60 = 96.666%
at t=118, the 1h error rate is 2/60 = 3.333%
At this point, our 1h error rate is below the alerting threshold, and so the alert is no longer in the firing state, 58 minutes after the end of the event.
at t=120, our 1h error rate finally drops back to 0%, as no errors have been observed in the last hour.
