Consulting DevOps from the Ops side

Question

Background

One of the org units in my company is doing application support for a relatively large client; that is, they're keeping some of their business applications running 24x7, doing anything from the root account upwards (no hardware/network). Things like keeping volumes, RAM, CPU sufficient, routing bug tickets where they need to go, backup&recovery, all that stuff. All the applications (many) are in a quite stable state, little development is going on.

It's 100% classical, i.e., 100% Ops. The occasional contacts with development are laden with the usual problems (mistrust, prejudice, etc.).

The customer decided that it wants to be modern, and is in the process of introducing "DevOps" - without having a clear understanding of what that actually means. They just know that they don't want to have one large entity running a large cluster of servers, not really knowing the software indepth, but that individual teams should build and support their own applications. Note that there is no container technology or anything like that - they are getting into OpenShift, but softly-slowly. Everything is quite classical... trouble ticket systems, multi-tier support levels, a great many tickets solved regularly, but all of them very easy in itself (like volumes running full etc.), very occasional difficult problems that need developer input.

So I'm on a taskforce of ours that's supposed to come up with ideas. I have quite some experience consulting Dev teams to be more "DevOps"y - i.e., more automation, more tooling, keeping dev/test/prod environments identical, that kind of thing. From their point of view, it's pretty clear how to proceed. But in my current example, it's not quite clear to me what to suggest. They do have problems, but more in the "Ops" side (like finding enough good staff to man the 24x7 shifts and so on). There does not really seem to be a "Dev-Ops" problem to be solved.

Question

In an Ops-heavy department with only a small amount of development going on, what are some DevOps practises, culture changes, process changes etc. that are very likely

• beneficial, no matter what they are doing so far
• relatively easy to introduce
• not that likely to induce fear and instinctive resistance (which there seems to be quite a case going on here, after the first talks)

For example, in a Dev team, I'd show them what to do with containers, or 12-factor, or ... and what concrete problem it solves, and this usually shows them that there's not much to fear, and that there are real benefits to be reaped. From there on, it goes as far as they want to take it.

The only objective problem I saw so far was that they really have little to no insight in the software, i.e., they mostly cannot even read the source (due to lack of knowledge) which would, in some cases, help in solving/routing bug reports etc.

The main subjective problem seems to be that they think real bad about developers. (According to them, those are lazy, picky, don't want to do 24x7, solve every ops problem simply by "re-rolling containers" etc.).

Answer 1

The key to presenting solutions is always to identify problems first; if the team doesn't feel like you're solving any problems, then they're going to dismiss what you're suggesting. The specific problems your Ops team is facing we don't know, but there are a number of common ones:

• Getting non-actionable alerts
• Spending lots of time in toil
• Version upgrades are a pain
• Deployments are a pain
• No visibility into what the apps are doing

and so on. Once you know what problems they're having, then you can target solutions that fix those.

The main subjective problem seems to be that they think real bad about developers.

Cultural changes are incredibly difficult to make (usually the easiest way to is hire people with the philosophies you want and gradually prune off the old people). But embedding operations engineers with developers can help this one immensely. I've worked in several small companies where there was just one engineering team (and I happened to work primarily on ops work), and so this idea comes very naturally to me, but for people who have been working solely on an operations team with little contact with developers for a long time, it can be a difficult adjustment. But move your desk to be with them, participate in their chat channels, eat lunch with them - this all helps both ops and devs realize that the other side are actually humans. This also provides better opportunities for ops to use their expertise to solve problems for devs that otherwise wouldn't've surfaced (say, they're trying to do something and you provide a quick shell script that solves it much faster than the manual way they were going with), and that builds trust and emotional budget that you can spend when you need something from them.