What are some methods to measure the ROI for DevOps?

Question

DevOps is complex, and involves many non-deterministic aspects like culture and process.
What are some ways to measure DevOps initiatives for success?
How do you prove to a business that the investment they have made is returning (or saving) real dollars?

Answer 1

Great question! Most of us know investing in DevOps practices is a highly worthwhile pursuit for myriad reasons; we don't often justify DevOps on the impact to the bottom line alone, though.

Note: this is something of an opinionated question, and my answer is likewise, opinionated.

Tensibai wisely suggested we find the right metrics, and he used time-to-market as an example. This is a great big-picture approach.

As an alternative approach, my experience with bean-counters is that they don't need to--or necessarily want to--know the big-picture, they just want evidence of fiscal responsibility. They want to see a trend in the right direction.

Here are just a few fiscal wins:

• calculate the server costs saved by leveraging auto-scaling in the cloud
• for income-generating sites, extrapolate the cost-per-minute of downtime, then show improvements in MTTI and MTTR
• again, for income-generating sites, estimate the cost-per-minute saved by leveraging highly-available architecture based on past incidents
• if you've improved on your build and deploy pipeline, show that you've reduced regressions and errors in production caused by already-tracked faults
• if you've made improvements to developer test environments, or even the tools and configuration on developer laptops, look at commit histories to see if new engineers are making their first contributions sooner after joining
• perform a full cost comparison between cloud and on-prem, much like Gitlab did recently, to justify your infrastructure spending (a.k.a. savings!)

Showing that you're money-conscious and you have a few clear wins is often sufficient. I've certainly missed some obvious examples; feel free to add comments below.

Answer 2

The key metric for a DevOps pipeline is Cycle Time (also called Lead Time). This is the time it takes for a change (or a change request, tracking all the way to the inception of the idea). The best illustration of this concept I know of is from the book "The Goal", which talks about it in a manufacturing context.

Deployment Frequency is useful as well. We want deployments to be frequent in a DevOps pipeline. There is no magical "1 day is good, 2 days is bad" measurement; this will need a historical context on your project to be meaningful.

Deployment Size: Measured in however your developers measure work - user stories, story points, quatloos, whatever. Again, you want to see trends over time, not absolute value.

Between frequency and size there is a story to tell. Are our releases becoming more infrequent and larger? why? Are they becoming smaller and more frequent? Again, why?

Explaining whether the frequency/size trend is good, we will also need Percentage of Failed Deploys. Uncovering the 'why' in those three metrics will tell you a lot about the health of the project.

My personal favorite, although it is a vanity metric, is Time for a Trivial Deploy. If you found the smallest possible thing worth redeploying the entire site over... perhaps a typo in the CEO's name... how quickly could you go from the panic phone call to a deployed site? I say 'vanity' because it really isn't that predictive beyond what the other metrics above discuss, but makes me feel good when I like the value.

If we get into monitoring, there are a bunch of different things we can track... from all-encompassing things like 'Uptime', to really low level things like the time spent regenerating custom HTML on a request-response cycle... but those aren't specific to instituting a DevOps culture.

These don't directly tie to dollars... doing so will require more knowledge about your org than I can offer in a forum like this; but they are the key to BEGIN to answer that question. Once you know you are able to regularly release work into production as a non-event, you can begin to see how much effort you were wasting before. As the book "The Goal" teaches (about manufacturing pipelines - it's relevant), optimizing locally can look like you're saving money, but ultimately, it just creates value that is tied up in inventory (undeployed features).

Beyond this advice, you should take a look at the State Of DevOps Report for the past few years. This is full of measurements about real world projects that you could emulate.

Answer 3

Captain obvious : by reducing time to market and defects on releases.

To extend on this one liner, the usual pitfall is being an organization change without any reference.

Engaging a culture or organization change implies some expense to train and introduce people to this new method, this have a cost in training but also imply a loss in productivity as people in a train session won't produce anything. This is the investment part of the cultural change.

To measure a ROI you have to first find some relevant metrics which should be improved (understand costly, either expensive or source of loss of profit). This could be a shorter time to market, less patch after each release, a better customer engagement within your product. Relevant metrics will be highly dependent of your product(s).

Measuring a ROI (how fast you have covered the training expense) implies that you can factually present an evolution on those metrics, so before engaging any change you must have defined those metrics and measured them in an objective way.
Once you have a real evolution to show you can tell if you did improve something in a way which has covered the training expense and become more profitable than it was before.

The usual pitfall is to engage the change before having defined any metrics and thus evaluating the ROI on a feeling and not on factual datas.

Productivity can be a metric, but its measurement is usually very hard to do in an objective fashion and should not be a first class metric for this kind of study.

Answer 4

Late to the game here but thought I'd chime in.

You can't manage what you don't measure, so for starters, here are the key metrics devops teams should be tracking for incident response:

• Uptime % : total % of time available = [total time - downtime] / [total time]
• MTTR : mean time to resolution = [downtime] / [# incidents]
• MTTA : mean time to acknowledge = [total time to acknowledge] / [# of incidents]
• MTBF : mean time between failures = [total time - downtime] / [# of incidents]

These metrics give you a high level health check of your operations, and help you identify where you need to dig in further.

Take a look at the whiteboard animation here for a more in-depth look on the topic.

Once you know your metrics, you can bump them up against the cost of downtime. You can start building out your team's ROI from there and set some quantitative metrics for continuous improvement.