I'm trying to understand the best way to capture data to start measuring Mean Time To Repair (MTTR) metrics, and I need to wrap my head around how "rollback" impacts MTTR positively or negatively.
Assuming that solid monitoring is in place, code is deployed that causes an incident that is detected rather quickly (low MTTI). At the point of identification, there are two main possible paths forward (yes, I'm oversimplifying for purposes of discussion):
Rollback the deployment, returning stability quickly, but without the intended features in production.
Roll-forward with additional changes that resolve the incident and keep the intended features live.
In this scenaro, MTTR is pretty darn low, given that site stability can come back pretty quickly. That said, the intended outcome of the change isn't live, and thus the code/feature/change is still stuck in process. If a goal is low MTTR, it seems to incentivize roll-back as a recovery mechanism.
In this scenario, MTTR is strictly measured by how long it takes the expected code/feature/change to be working properly in production. Even if I rollback, until my "fixed" code change goes into prod, the MTTR timer is still running. In this case, MTTR seems tied to business outcome stability instead of just pure "hey, things are stable."
Now, the answer may be as simple as MTTR not being used as a metric in a vacuum, but rather in conjunction with Change Failure Rate - a super-low MTTR caused by frequent rollbacks could point to a sky-high Change Failure Rate. That said, there's something that doesn't seem right to me in the idea of divorcing the MTTR measurement from business outcome.
I may be way overthinking this, but I'm curious how others are measuring MTTR and what the end point-in-time is for "recovery." Are you using it simply as stability, or do other factors come into determining what "recovered" means?