You are asking the right questions and hitting on a very important aspect of these metrics. Individual metrics are not super helpful and measurements need to be defined. Having worked with these measurements across small and large organizations, let me walk you through my thought process.
- Metrics - Some sort of calculation on one or more measures
- Measure - An amount of something using a specific unit
- Change Failure Rate (CFR) - The percentage of deployments causing a failure in production
- Mean Time to Recover (MTTR) - How long it takes an organization to recover from a failure in production
This conversation could include Service-Level Indicators (SLI), Service-Level Objective (SLO), and Service-Level Agreement (SLA), however, for this purpose I will assume we have measurements (SLI) to ensure we are meeting our goals (SLO) and not breaking our promise to users (SLA).
Before getting into the details, the main question asked is:
What are we trying to measure?
At the end of the day, any organization exists to create value, these metrics are meant to track any interruption in the flow of value from the organization to its consumers / users.
As an example, if we wanted to measure how efficiently the organization adds new value to users, Lead Time for changes is a great way. How long does it take new code or business ideas (depending on if you would like to include non technical business processes) to get from creation / ideation to the user? Lead Time includes CFR and MTTR, both of these can interrupt getting value to the consumer and increase Lead Time.
Using the value creation lens, let's look at CFR first.
What does the organization consider a failure in production?
I have typically used it as an event needing immediate fixing, on the call during the production deployment. The service we deployed needed to be rolled back, a hotfix needed to be pushed immediately to correct an issue discovered during the deployment, or something equally catastrophic. These are all interruptions to value reaching the user. A bug not presenting itself for a few days, would not create a Change Failure with this definition. However, if the bug began to trip indicators (SLI) (example, degraded speed or experience for the end user), exceeds our objectives (SLO), or even breaks or promise to users (SLA) then an incident is created and it will bring us to your next measurement, MTTR.
Assuming the bug mentioned creates an incident, MTTR measures how long it takes to fix it. If the fix is a roll back for the original release, or a hotfix of the release, it really does not matter. The measurement is how long it takes to correct.
Pulling this together.
- Determine what the measure is.
For CFR, what does the organization consider a production failure? I provided a few examples, but all organizations are free to create their own measure. For MTTR, the measure is time, hopefully from issue creation to correction, sometimes we need to settle for incident discovery to correction.
- With the Measure, track the Metrics.
How many failed production releases are there? How long, on average, does it take to correct production incidents?
- With the Metrics, track and, hopefully, improve the flow of value from the organization to the consumer.
If a deployment is successful, it should not trip the measurement for Change Failure, however, if there are bugs, their interruption to value will be caught with Mean Time To Recovery.
These measurements and metrics (Lead Time To Change, Mean Time to Recovery, Deployment Frequency, and Change Failure Rate) are meant to tell a story in aggregate.
Assuming weekly deployment, if an organization has 0% CFR, but a 14 Month MTTR with incidents occurring every week, then the flow of value is constantly interrupted and the amazing CFR score means very little. Or, if the CFR is 50% and the MTTR is 2 Mins, then the high CFR looks horrible, but the actual interruption to users is only 2 Mins * bugs per week. It really takes both, or all four of the metrics, to create an accurate picture of the real state of the code and deployment process.
Hopefully this helps. Working through these measurements and metrics is a journey, probably with no conclusion, with a goal of improving the amount of value an organization can create for its consumers.