Definition of change failure rate and mean time to recovery DORA metrics

Question

There has been a lot of talk in recent years about the Accelerate book and the DORA metrics.

As the DORA survey was looking at a wide variety of organisations, teams and cultures all with potentially very different current processes and technologies, the questions asked were fairly general and probably leave a fair bit of wriggle room for teams to interpret things their own way. Which is all fair enough.

However, I'm specifically trying to understand what change failure rate and mean time to recovery are trying to measure. Are they just trying to measure if the release process is good or if all of the changes in that release are good?

If I do 10 releases in a row and the release process works smoothly for that and I get no downtime, no manual release tweaks, no alerts, etc...then it looks like I've got a 0% failure rate. But what if there was a relatively subtle bug in that very first release and it doesn't get picked up for 3 days so there is a bug fix going into the next release to fix it. And of course the next release might also have some bugs and so on, so you could say I've got a 100% failure rate due to the bugs.

My instinct is that I should have a high failure rate in this scenario because although the process on the day of release looks good, it the software itself is not good. Just because you have not tested your release before or immediately after release, it doesn't feel like you should get away with claiming no failures.

So if I do get a bug that isn't fixed until the next release in say a week's time, then my mean time to recovery might be a week...unless I did a rollback of that initial release?

What are the common interpretations/implementations of these metrics?

Answer 1

You are asking the right questions and hitting on a very important aspect of these metrics. Individual metrics are not super helpful and measurements need to be defined. Having worked with these measurements across small and large organizations, let me walk you through my thought process.

Definitions:

• Metrics - Some sort of calculation on one or more measures
• Measure - An amount of something using a specific unit
• Change Failure Rate (CFR) - The percentage of deployments causing a failure in production
• Mean Time to Recover (MTTR) - How long it takes an organization to recover from a failure in production

This conversation could include Service-Level Indicators (SLI), Service-Level Objective (SLO), and Service-Level Agreement (SLA), however, for this purpose I will assume we have measurements (SLI) to ensure we are meeting our goals (SLO) and not breaking our promise to users (SLA).

Before getting into the details, the main question asked is:

What are we trying to measure?

At the end of the day, any organization exists to create value, these metrics are meant to track any interruption in the flow of value from the organization to its consumers / users.

As an example, if we wanted to measure how efficiently the organization adds new value to users, Lead Time for changes is a great way. How long does it take new code or business ideas (depending on if you would like to include non technical business processes) to get from creation / ideation to the user? Lead Time includes CFR and MTTR, both of these can interrupt getting value to the consumer and increase Lead Time.

Using the value creation lens, let's look at CFR first.

What does the organization consider a failure in production?

I have typically used it as an event needing immediate fixing, on the call during the production deployment. The service we deployed needed to be rolled back, a hotfix needed to be pushed immediately to correct an issue discovered during the deployment, or something equally catastrophic. These are all interruptions to value reaching the user. A bug not presenting itself for a few days, would not create a Change Failure with this definition. However, if the bug began to trip indicators (SLI) (example, degraded speed or experience for the end user), exceeds our objectives (SLO), or even breaks or promise to users (SLA) then an incident is created and it will bring us to your next measurement, MTTR.

Assuming the bug mentioned creates an incident, MTTR measures how long it takes to fix it. If the fix is a roll back for the original release, or a hotfix of the release, it really does not matter. The measurement is how long it takes to correct.

Pulling this together.

1. Determine what the measure is.

For CFR, what does the organization consider a production failure? I provided a few examples, but all organizations are free to create their own measure. For MTTR, the measure is time, hopefully from issue creation to correction, sometimes we need to settle for incident discovery to correction.

2. With the Measure, track the Metrics.

How many failed production releases are there? How long, on average, does it take to correct production incidents?

3. With the Metrics, track and, hopefully, improve the flow of value from the organization to the consumer.

If a deployment is successful, it should not trip the measurement for Change Failure, however, if there are bugs, their interruption to value will be caught with Mean Time To Recovery.

These measurements and metrics (Lead Time To Change, Mean Time to Recovery, Deployment Frequency, and Change Failure Rate) are meant to tell a story in aggregate.

Assuming weekly deployment, if an organization has 0% CFR, but a 14 Month MTTR with incidents occurring every week, then the flow of value is constantly interrupted and the amazing CFR score means very little. Or, if the CFR is 50% and the MTTR is 2 Mins, then the high CFR looks horrible, but the actual interruption to users is only 2 Mins * bugs per week. It really takes both, or all four of the metrics, to create an accurate picture of the real state of the code and deployment process.

Hopefully this helps. Working through these measurements and metrics is a journey, probably with no conclusion, with a goal of improving the amount of value an organization can create for its consumers.