What is the proper relationship between rollback/rollforward and MTTR metrics?

Question

I'm trying to understand the best way to capture data to start measuring Mean Time To Repair (MTTR) metrics, and I need to wrap my head around how "rollback" impacts MTTR positively or negatively.

Scenario 1

Assuming that solid monitoring is in place, code is deployed that causes an incident that is detected rather quickly (low MTTI). At the point of identification, there are two main possible paths forward (yes, I'm oversimplifying for purposes of discussion):

1. Rollback the deployment, returning stability quickly, but without the intended features in production.

2. Roll-forward with additional changes that resolve the incident and keep the intended features live.

In this scenaro, MTTR is pretty darn low, given that site stability can come back pretty quickly. That said, the intended outcome of the change isn't live, and thus the code/feature/change is still stuck in process. If a goal is low MTTR, it seems to incentivize roll-back as a recovery mechanism.

Scenario 2

In this scenario, MTTR is strictly measured by how long it takes the expected code/feature/change to be working properly in production. Even if I rollback, until my "fixed" code change goes into prod, the MTTR timer is still running. In this case, MTTR seems tied to business outcome stability instead of just pure "hey, things are stable."

Now, the answer may be as simple as MTTR not being used as a metric in a vacuum, but rather in conjunction with Change Failure Rate - a super-low MTTR caused by frequent rollbacks could point to a sky-high Change Failure Rate. That said, there's something that doesn't seem right to me in the idea of divorcing the MTTR measurement from business outcome.

I may be way overthinking this, but I'm curious how others are measuring MTTR and what the end point-in-time is for "recovery." Are you using it simply as stability, or do other factors come into determining what "recovered" means?

Answer 1

Yes, MTTR is/should always be tied to the business outcome: if things are not stable the very business is at risk.

The fact that the expected code/feature/change is still stuck in process in scenario 1 is irelevant: the feature is not stable, so it doesn't bring new business, rolling back is the best you can do at that time from the business prospective.

The rollforward is a gamble: keeps the business at risk waiting for a potential fix that in fact has statistically lower changes of success (due to the instability it will always be rushed compared to the change that caused the instability in the first place without even having such pressure on it). The rollforward is a yet another version of the code which hasn't been checked before.

If you want to keep the MTTR low you rollback immediately, without debate. This removes the business risk and gives you a chance to check that the fix is actually working before attempting to deploy it. I'd strongly suggest making it a policy as yes, almost always there will be someone asking for a fix instead of the rollback and calling a meeting to negociate/decide on it - all while business remains at risk.

Side note: if you're concerned with a high Change Failure Rate then I'd suggest checking the the actual rollback rate instead of deriving it from a low MTRR. Maybe you'd like to add a gate check before deployment for the most frequent failures. If you have such check already automated - why not include it in the CI verification? If you don't have one - maybe its time to start thinking about it? :)

Answer 2

The mean time to recover has an implied subject - the mean time to recover what? Defining this is key to using the metric effectively.

Are you recovering the general availability of your production website? Are you recovering the functionality of a particular feature that has a bug in it? Once you know what you're actually trying to measure, it's much easier to measure it!

The general thrust of your question seems to actually be surrounding the competing goals of shipping features and maintaining reliability, which is an ages-old battle. Traditionally it is developers' jobs to implement new things, and sysadmins' jobs to prevent things from breaking, and this leads to departmental conflict, as change tends to cause breakage. One of the philosophies oft associated with DevOps is the idea that developers and ops engineers should work closely together so as to ease this tension.

You may also be interested in Google's approach to that problem, which is to have "error budgets" for development teams to spend; once they've penalized stability too much, they must spend the rest of the quarter only working on stability. Along with this, the site reliability engineers have available goals, and if they over shoot, they are encouraged to let more changes through; the idea here is that their goal must not simply be to maintain reliability as high as possible, as then they'd be motivated to fight change in every situation.