13

I am trying to drive good behaviors within a DevOps transformation programme, to support this I am looking at identifying actionable metrics around the operations disciplines:

  • Problem and Incident Management
  • Capacity Management
  • Change and Release Management

To be absolutely clear, these are functions that used to belong to the operations organization and are now owned by the Agile/DevOps organization. There are existing KPIs that drive bad behaviors are:

  • Time to Root Cause Analysis completed:
    • Drives incomplete RCAs just to get them into the system on time.
  • Test execution duration:
    • Disables long running tests, regardless of their business value.
  • Average utilization of cloud services:
    • Encourages over-commitment of compute resources, resulting in slow response times

What Key Performance Indicators can be used to encourage good behaviour in a DevOps Programme?

4
  • 1
    You've found the inherent problem with all KPIs. People will work to maximize the performance indicators instead of maximizing actual performance. Metrics give people a score to run up, and they will, even at the cost of doing their jobs well.
    – Adrian
    Mar 31, 2017 at 20:30
  • @Adrian I agree, however there are certain KPIs, such as cycle time, that are inherently difficult to game. Mar 31, 2017 at 20:34
  • True. Still, even those that are hard to game will lead to optimizing for the KPI, which may be sub-optimal in general, or simply favoring those KPIs that can be gamed. I've found very few ways to measure DevOps performance automatically with metrics; it's mostly subjective and requires personal observation and engagement.
    – Adrian
    Mar 31, 2017 at 20:38
  • That's not DevOps, it's ITIL haha
    – Gaius
    Jul 17, 2017 at 9:47

3 Answers 3

12

I don't think there are any "universal" DevOps KPIs. For example, velocity is great, unless it's not a key driver for your business. Amazon cares a lot about velocity because they have a massive retail operation. That's less important for a small app with 100 users.

This begs the question: how do you select the best KPIs relevant to your business? That's a research and discovery process that involves your entire Enterprise.

What do you care about?

  • Quality
  • Reliability
  • Maintainability
  • Velocity
  • Process Improvement
  • Service Levels

What keeps your business stakeholders up at night? What determines whether you make money this quarter or not? The list above might include some of those things, or it might not. Make your list, then figure out how to align incentives across every department to achieve them.

Incentives drive behavior, so decide collaboratively on SMART goals. Pick two or three items off your brainstormed list, and start a measure/fix feedback cycle for each. Don't pick too many at once- you're more likely to succeed by focusing hard on two or three things.

3

DevOps does not have any KPI. It would be like asking what are the KPI of Love. But some of the things you mentioned (Problem and Incident Management, Capacity Management, Change and Release Management) do have good KPI, some of which are based on the theory behind DevOps.

In general, for any business process, you can create a Value Chain Map describing how value flows from Customer, through the enterprise back to Customer. The entire loop always has to start and end with Customer, but sometimes, for a service organization, the Customer can be internal. The throughput of value through such chain can be a good way to design your KPI in a tamper-proof way. Measuring any KPI in any individual link of the value chain only makes sense as long as that particular link is the bottleneck of the process and you try to exploit or elevate the bottleneck.

A common problem with KPI is when it starts halfway through the chain. For example, a Change and Release process often starts with developers and ends with deployment. This process excluded:

  • Customer having a problem
  • Support team identifying the problems
  • Product team defining the problem for backlog
  • Solutions team customizing the deployment for the customer
  • Customer realizing value from the solution

The problem is that measuring a cycle time alone could lead to two major problems:

  1. The bottleneck is in any of the excluded parts mentioned above and your customers are not realizing the value and you are not realizing the revenue proportional to the speed of your cycle time. So while your engineering is excellent, your business suffers.

  2. The disconnect from Customers will make your release cycle spin on empty - not producing any value, despite change being done - or even counter the needs of your Customers and the work being done could have negative business value.

Another problem with KPI is that while starting and ending with a customer it might not actually measure the value to the customer. A good example would be a Problem and Incident Response process and MTTR (Mean Time To Repair) as KPI. Does the problem even affect anyone? What is the value to customers? You could have excellent MTTR of 3 hours over 100 incidents. But if most of them were internal, with no or minimal impact to customers and resolved in minutes, while the one large incident with huge customer impact took 3 days to handle, the business value is lower than if you had 1 day MTTR, because you ignore most of the internal incidents, but you responded to the huge customer impact incident within 1 hour.

NOTE: For an internal customer, in case of the support team business process, the derived value is not the value of the work to the internal customer, but the value gained by the business in unblocking the internal customer in their own business process. Unblocking a team that is a bottleneck in their own process derives higher value than unblocking a non-bottleneck team or individual. All KPI of such support team need to include the business value in their calculation.

1
  1. Deployment frequency
  2. Speed of deployment
  3. Deployment success rate
  4. How quickly service can be restored after a failed deployment
    And finally,
  5. DevOps Culture, which actually can’t be measured
1
  • 5.DevOpsCulture, which actually can’t be measured => make an anonymous questionaire for everyone involved even slightly and ask them how they feel about all of this. This surely will tell you if you have a process that is lived by your people, or if many people are in truth half-way out of the door...
    – AnoE
    Nov 23, 2017 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.