How to calculate burn rate for SLOs?

Question

I've read the Google SRE book a few times but I need some clarifications on exactly how to set up the burn rate and understanding how long it'll take to trigger an alert.

Most of my questions are specifically from this section on the book: https://landing.google.com/sre/workbook/chapters/alerting-on-slos/#4-alert-on-burn-rate

1. In Table 5-4, the error rate for 99.9% SLO for burn rate of 1 is equal to 0.1%. I just want to confirm - this 0.1% is coming from 100% - 99.9% where 99.9% is the SLO. Does this mean if some service has a ridiculously lower SLO, (say 90% SLO), then their error rate for burn rate of 1 would be 10%. Am I interpreting this properly?

2. Couple sentences below, the book says

For burn rate-based alerts, the time taken for an alert to fire is (1-SLO/error ratio) * alerting window size * burn rate.

So if my SLO is 95%, and my error rate is 1 (let's assume all the requests coming up in last 1 hour are errors). Let's assume my burn rate is 1. If I plug these values into the formula, I am getting,

(100-95/1) * 1 hour * 1 = 5.

This is where my confusion is. Is this 5 hours? Do you convert 1 hour into 60 minutes to get the minutes? Is that how long it'll take Prometheus to fire the first alert?

Furthermore, isn't the detection time too late if it will take 5 hours to get an alert? Maybe some concrete examples on how you use this formula to calculate some real values would be really helpful.

3. The next formula,

The error budget consumed by the time the alert fires is: (burn rate * alerting window size)/period.

Just want to clarify this - if my burn rate is 1 and my alerting window size is 1 hour, this means, within 5 hours, I would have consumed, (1 * 1)/5 = 20% of my error budget.

Is this right?

4. The table 5-8 has recommendations to send an alert if the long window is 1 hour and the short window is 5 minutes and if the burn rate is 14.4 and the error budget consumed is 2%.

In the graph above, it shows that the error rate for 5 minutes (10%) is higher than the error rate for 1 hour. Why would their error rates be different if the burn rate is same = 14.4? I'm having difficulty understanding this bit.

It also says it will take 5 minutes to alert based on this information. Will this be true for any SLO? Or is it only true for 99.9% SLO?

5. And lastly, just wanted to clarify: The examples in the chapter use this following recording rule, which is for ratio of errors per request.

job:slo_errors_per_request:ratio_rate1h{job="myjob"} > (14.4*0.001)

If I want to something similar for identifying latency - i.e. ratio of all requests which have a latency of more than 2 seconds, then this would be the following PromQL:

(sum by (job, le) (rate(latency_quantile{job="myjob", le="2"}[1h])) 
/
sum by (job, le) (rate(request_count{job="myjob"}[1h]))) > (14.4 * 0.001)

Does this look right?

Bottom line, I wish this particular chapter, which the authors acknowledge that it has some complex implementations, had some more concrete examples especially around the formulae and the burn rate. Some examples specified in the tables and chart make sense. But some need a little more clarification to understand the nuances (e.g. specifying that a burn rate of 14.4 translates into 2% of the error budget and this is because when you divide by 30/14.4, you get 50 hours of error budget, and 1 hour is 2% of those 50 hours.)

Answer 1

1. Yes, you are interpreting that correctly.

2. When you express your SLO as a percentage, you should express your error rate as a percentage too. So, in you example your error rate of 1 is 100%. That would make the equation:

   (100-95/100) * 1 hour * 1 = 0.05 hours = 3 minutes
   

3. The period in that equation is the reporting period that you have chosen for your SLO. You have not mentioned what time period you chose. The examples in the workbook are 30 days. If you use a 30 day (720 hour) period in your example, you would consume approximately 0.14% of your error budget in the 3 minutes it takes for the alert to fire:

   ( 1 * 1 hour ) / 720 hours = 0.14%
   

4. The rates are different because they are measured over different time periods. (Note that the graph is using a logarithmic scale, so the actual error rate and the peak of the 5m measurement is 15%, not 10%.)

   • Consider what you would measure one minute after the errors started:
     • The 5m measurement would see 4 minutes of no errors plus one minute of 15% errors, which gives an overall error rate of 3%.
     • The 60m measurement would see 59 minutes of no errors plus one minute of 15% errors, which is just an overall error rate of 0.25%.
   • Then 5 minutes after the errors started:
     • The 5m measurement would see 5 minutes of 15% errors, so the overall rate would be 15%.
     • The 60m measurement would see 55 minutes of no errors plus 5 minutes of 15% errors, so the overall rate would be 1.25%. This is actually still too low to trigger the alert, so it will actually take more than 5 minutes to alert.

   • Using the equation from before, the actual time taken to alert is 5.76 minutes:

     5.76 = (0.001/0.15) * 60 * 14.4
     

5. Your PromQL seems to be looking at the good events instead of the bad events, as the le="2" selector is for the bucket counting latencies less than or equal to (le) 2 seconds. You may want to create a separate "good_latency_rate1h" recording rule for the good events, and then use something like "(1-good_latency_rate1h) > (14.4*0.001)" for your alerts.