How can I persuade developers on my team to embrace "You build it, you run it"?

Question

How can I persuade developers on my team to embrace "You build it, you run it"? By that, I have this quote from Werner Vogels in mind:

Giving developers operational responsibilities has greatly enhanced the quality of the services, both from a customer and a technology point of view. The traditional model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. Not at Amazon. You build it, you run it. This brings developers into contact with the day-to-day operation of their software. It also brings them into day-to-day contact with the customer. This customer feedback loop is essential for improving the quality of the service.

I'm specifically thinking of a set of developers that:

• Were hired into a developer role, with little/no mention of ops-related tasks.
• Traditionally have "thrown code over the wall" to an ops team.
• Traditionally have a 9-5 work schedule, and are actively hostile to the idea of "pager duty", participating in disaster recovery, writing post-mortems, etc, especially outside of normal business hours. (Note: I only have very infrequent outages in mind for this; I am not proposing that we add after-hours customer support to this team's workload.)
• Are not currently responsible for writing/supporting monitoring or alerting on their applications.

Suppose there is a team that is rapidly developing new cloud micro-services with a profile that is getting to be such that handing these services off to an ops team is sub-optimal because they can't keep up in regards to gaining deep knowledge of the services that is required to effectively manage and monitor them. "You build it, you run it" would work better for this team because tasks could delegated to each responsible team member. So this team would begin taking part in designing infrastructure, monitoring/alerting tools for the services, and (very infrequently) responding to outage events.

I am specifically interested in methodologies, backed up by real world examples. How this been successfully implemented in other workplaces, and if there any canonical steps to follow while implementing this? Any links to write-ups that can support answers would be very helpful.

Answer 1

I think the easiest way is to change their performance goals so they are based off reliability as well as delivering code. Sell it as the company cannot succeed without both so why should the developers only be measured on one? The best way for them then to meet their performance goals is to be involved in operations.

Ultimately you need to convince them this is the best way for the company to succeed and therefore for them to succeed. That's hard and you can't expect them to be onboard from the start. They need to be sold on the value too.

Answer 2

When it comes to affecting business culture, the best way is probably via the well-known "boil the frog" method. You have to introduce these tasks to developers slowly, because I know I myself (as a dev) would balk at having all this new responsibility at once.

First, start out by introducing one or two new tasks only to be performed during normal business hours. They need to learn how to do devops which could be quite the learning process for a (to this point) code-only dev and might require some supervision. They will also likely be hostile to the idea of changing their work-life balance since you mention they're used to 9-5. At this point, record data on the new processes for use later (have them write this code, data is always useful).

Later as you're running out of new devops-y tasks to introduce (so the first, second, and fourth bullet points are almost complete), bring the first tasks you've introduced up as candidates to be performed outside of standard work hours. You may see some backlash at this and you might even see some attrition depending on how strongly you push this and how heavily the work-ends-at-five culture is ingrained. To defend against this, hopefully your data supports the idea that work beyond standard hours will be rare, only occur in extreme situations, and will greatly benefit both the business and the customer. If your data doesn't support this, then you'd better be ready to deal with the consequences of this choice.

Even with data, it may still be easier to have the developers write the monitoring/alerting code (so they become devops but still mainly dev) and keep the alternate ops team as front-line support (as a few others have suggested). Like I said, small changes are important to avoid backlash. Integrating work for devs beyond standard hours will be challenging, as they may know they can look elsewhere for employment if they don't like it since the market for devs is strong right now, especially if they already had devops skills. Caveat emptor!

Answer 3

IMHO You build it, you run it should not be taken literally. To begin - it almost sounds like a punishment ;)

No single person or even small developer team can reasonably support a tool or a toolset used in the software development process (i.e. in production!) for long periods of time. Been there, done that :)

Support duties tend to expand as the tool(set) customer base grows. If assuming those duties exclusively, the development team could end up doing mostly support, with little/no time left for development down the road. Effectively a dead-end - how many developers would want to stick around in such environment?

Having a professional 1st line of support team is crucial to prevent frustration, stress and other effects of long-term exposure to support duties over your developer team members.

The 1st line support team would, of course, fallback to the developer team (again, team, not single-person!) for issues they cannot cover directly. Yes, it may be difficult at first, but thing will get better. It should be a collaboration - that's part of what DevOps is about.

Answer 4

Looking outside of DevOps specifically, if you are talking about large (ish) enterprise environments then the SAFe methodology has a fairly good way of encouraging this kind of behavior.

Essentially it boils down to a few key stages:

• projects start as release trains. The intent (and the expectation of whoever is funding it) is that it will be long running. I'm talking years long, none of this "stand up a team for 3 months then stand them down" business.

• over the course of a project the release train will inevitably amass more people as more of both new project requirements based on newly realised business opportunities as well as the ongoing tax of maintenance style work.

• I almost always see the trains run their first increment as 100% project/change teams, 2nd increment they allocate a percentage of time to Run work. 3rd increment management realise they are about to have a problem and starting adding teams to try and handle the Run overflow to give their now seasoned developers and engineers time to keep working on new requirements.

• if the right balance is struck of project teams being able to keep delivering project outcomes without being bogged down to much in maintenance work and new teams who join the train aren't immediately expected to just be 100% support teams, but instead take on a fair portion of the change work that was going to be handled then you end up with delivery teams who are owning the products they are building by default, it doesn't need to come at them all at once that now they are a Run team, instead it just slowly integrates into their day to day activities.

Where this model obviously has some weaknesses is in the pricing for a business. Generally I'd expect to be paying a lot more for a change resource than a run resource, usually run contracts with major vendors are fixed price and the intent is that they make their money on continuous improvement and therefore cost savings.

That being said, it isn't all that difficult to consider the change teams stood up as part of a release train to simply be delivering continuous improvements. If there is something they can build or do which will improve their ability to focus on new project deliveries and be less concerned with "business as usual" work then that should be backlogged and prioritised based on the perceived benefits by whoever holds the money for the release train.

Answer 5

This ultimately depends on the size and structure of the company. There are really three stages that your company can be in:

1. The startup stage (under 150 engineers). Of course, the developers need to run their own software. Everyone does so in a startup. You might not even have operations team to begin with, but even if you do, it is small and the speed of progress is so fast that there is no way to pass the knowledge required to run it successfully on another team quickly enough for them to be successful.

2. The medium sized business (over 150 engineers, but a single operations team). At this stage the churn in the company starts to be too high, the engineers who build software do not necessarily stick around also to run it. You don't know everyone anymore and it is hard to communicate effectively for everyone to be in operations. It would start to turn into chaos. At this stage you want to turn to the Google model, where every team has to operationalize their software, but not necessarily operate it. They will operate it at the start, but a big part of building software is to reduce the cost of operating it to a point, where the load is small enough that the operations team can sign on running it. Only then it is considered done.

3. Large enterprise with multiple business units, (where each has its own operations team): At this stage you can turn back to the Amazon model, where every team essential develops and operates their own services. Each of the business units has to be small enough for everyone to know each others, so under about 150 engineers and you operate each essentially as a startup. Amazon has each AWS service operating more or less separately and it works for them.

You have to figure out which stage your company is in and/or moving into and act accordingly. There is no "one size fits all" solution.

Answer 6

My take on this (if I was faced with such commandement, or whatever you'd call it), would be something like "What else would you expect?!?!". Because, accidently:

• I wouldn't even be able to live without it, and
• I love to eat my own (dog)food.

Let me explain a bit further ...

My business / hobby / passion is scm, more specifically in mainframe environments. And wherever I go (to finetune stuff to fit the customers needs), the very first requirement the I impose (in my contract), is that any tuning done to the system we implement, is via that very same system. And by doing so (true, that takes like a few hours, say half a day at max), we get these benefits from it (incomplete list):

• I hardly document anything I did to get the system going. If any questions are asked, I just query the system (and teach the customer how to query the system without my help).
• If I get called in in X months/years (to upgrade to a new release?), I want to know (remember) what "I" have been doing before (and the only thing I trust is what the system tells me, aka reminds me about).
• I only need to ask the customer once about how to do specific things in their site (naming conventions are hard to remember), or to convince them to grant required permissions to "the system" (not to me ...).
• You're continuously QA-testing your own system ... in production. Chances are that you'll experience bugs, logical errors or missing features (and what not) yourself. And those are extremely motivating to get them addressed ASAP ... eg to become more productive.
• ... and if you'd want I can add another dozen of reasons ...

However, before you try this at home yourself, be aware of some pitfalls to avoid:

• you want everybody to join the party (use the system), because only 1 "exception" (aka the manager/owner of the company?) may ruin the party (you'd think you can trust your system, but then somebody did something without asking/using the system). Those case may cost you days to discover ...
• new users might complain that by using this (new) system, it takes them way longer to get their job done (as compared to whatever they used before). And wherever it makes sense, they'll use that as an excuse why they are late to complete their job. At that point, you better have upper management covering you, because otherwise your project/system may get the blame.
• make sure you do know your own system, and how to configure it to fit your needs. As a sample (in my case): you want to configure the system so that you use the operational environment to deliver to your experimental environment, and not the other way around ... I've seen that happening in the past ... using (abusing) the test system to deliver to highly secured environments.

Answer 7

Different organisations have different organisational structures, needs and cultures. Some developers think that operational issues are not their issues. However, a good development culture is one where developers care about the smooth operations of their systems, they also care about operation colleagues who need to support their system. Us developers used to carry pagers in case there were issues that operators cannot solve. Whilst there are clear team responsibilities, there is also a fuzzy line due to the complexity of developing and operating a system. If some staff do not appreciate that, then it can be difficult in some situations. I am not suggesting that development staff should be supporting ops all the time. If that happens too often, then there are issues somewhere that need fixing. Allow flexibility and try to look after each other.

Answer 8

What you propose is that every developer in your company should be a Jack of all trades, master of none. You should keep in mind that the Amazon model (constant startup mode) causes people to burn out from constant context switching, pager duty, and not focusing on what they do best. If you build an airplane or a formula 1 car, that does not mean that you should fly that airplane or drive that car because some people will do it better than you, while they would not be good at building.

When you run software like Jenkins, MySQL, Cassandra, Tomcat, do you expect the developers who developed that software to support your installation and every other installation in the world? Of course not, but you do expect to have tools and means to monitor performance, failures, memory consumption and other issues that might occur.

As Jiri Klouda explained in his answer Google uses that philosophy where it expects teams to operationalize their software, but not necessarily operate it. They also employ SREs while Amazon does not so developers have to pay the toll and do something they do not excel in.

Traditionally have a 9-5 work schedule and are actively hostile to the idea of "pager duty", participating in disaster recovery, writing post-mortems, etc, especially outside of normal business hours. (Note: I only have very infrequent outages in mind for this; I am not proposing that we add after-hours customer support to this team's workload.)

If you are certain that outages will be infrequent, are you ready to pay for the extra time when an outage occurs?

Other problems frequently happen in teams where developers are not just developers, but admins, customer support, pr etc. Because they do ops and dev, they do not make good documentation and there is a lot of tribal knowledge. They do not make it easy for some other team to run their software so when they leave the company it is hard to support their service. You should persuade your developers to make their software operationalized so anyone can run it with little or no help.