77

I have just been speaking to a DevOps guy who raised some really good points about the struggles of being a DevOps Engineer and feeling like a one-man army sometimes, even though he is in a team of 16 Engineers.

He wears lots of different hats, but sits in the Development team doing infrastructure work. He loves the cool tech that he gets to work with- loads of automation, cloud, containerization etc. But he struggles that he is the only person doing ops, in a dev team. He reports to the Development Manager, but works more closely with the Infrastructure Manager.

This seems to be the case with lots of DevOps professionals I speak to. What could be done to help DevOps Engineers feel less like a lone wolf?

Improve this question
2

8 Answers 8

Reset to default
41

My first thought is "why is the he the only person doing ops, on a dev team, especially when he gets to work with loads of automation?". I think there's an opportunity there to address the lone wolf syndrome; particularly in a dev environment, infrastructure-as-code provides a great framework for sharing the load. Ops people should be experts in determining and defining infrastructure needs for application, but they should also be able to build a platform to let dev roles do as much as they can independently.

It sounds like a silo within a team; old habits die hard. A coder may not feel comfortable spinning up and hardening a server, but in a pure devops model, they should have the tools to do so. An ops person in a devops team shouldn't responsible for delivering infrastructure for the app itself, but they should provide some insight into what is needed and some guidance on how the app developers can do it themselves. It's almost a meta-infrastructure model; ops engineers build infrastructure that can build infrastructure on demand when requested by the development team.

Consultation, communication, and blending of responsibilities are all crucial to the success of a devops team.

Improve this answer
2
  • 1
    Some of this is very soft software focused. I work with embedded software (or firmware or software running on/with specialist hardware) and many of the IaC models and tools do not fit well. Sometimes the DevOps guy is the only one who knows where that hardware is. I was 1 of the 4 out of ~60 engineers who could go find things in the lab. In those cases this answer is hard to implement. We did work out ways to let people power cycle units remotely but that was about all we could come up with. Any more suggestions would be welcome.
    – TafT
    Commented Nov 13, 2017 at 9:26
  • You’re right; I tried to frame my answer based on clues in the question (namely, the post mentioned automation); it applies less in your situation. That said, every situation is different, so every path is different. The principles of building automation and emphasizing self service and looking at the entire value steam still apply, even if the implementation differs.
    – Stuart Ainsworth
    Commented Nov 14, 2017 at 17:56
31

I think the first flaw is in this sentence:

He reports to the Development Manager, but works more closely with the Infrastructure Manager.

DevOps is a cultural shift aiming at removing silos. If silos remain, then this engineer is whatever you want to name call him; an engineer doing operational development, an automation expert, a developer automating infrastructure - but this engineer is not a DevOps engineer.
In fact, "DevOps Engineer" is not a real role, it's more a 'chapeau' as it can encompass developers, system administrators, QA testers and architects working in a common team.

A problem I often see is that people fall into the buzzword usage of DevOps, seeing it as a job title, but they don't really understand what DevOps is. By viewing DevOps this way, they often end up isolated and feel alone, blaming failures and shortcomings on being a "lone wolf" without managerial and organizational buy-in.

As you describe it this engineer is the only one doing Ops in a Dev team. That doesn't make him a "DevOps engineer". (Whatever that means in his organization) He's working in isolation because the job is presented as "DevOps Engineer" but it seems tha other people in his team doesn't wish to work on operations.

Let's be honest, there will always be ops and dev, the main idea behind devops is sharing the responsibilities such that there's no handoff of a product from dev to ops or just supplying a platforms by ops for dev. The primary goal is bringing more collaboration into a team. Calling this role a "DevOps Engineer" is breaking this idea by suggesting in the name you can do both at the same level of expertise - which is rarely true.

The first thing to do in my opinion would be to present the operational tools to the team and give everyone a basic knowledge on the tools, then transfer the responsibility of configure/code the operational tools to the whole team. The main idea behind this is moving from "the one doing all the ops things" to "the one which supports and give references implementations to the team".

This complements the other answers by providing something actionable in an simpler way as a first step than a management reorganization.

Improve this answer
2
  • 2
    One of the tough things to reconcile about a devops implementation is org charts. Silos typically form around management, and if you have a Dev mgr and an Infrastructure mgr, getting those teams to communicate sounds good, but there's a reluctance to consolidate. Culture is tough to change, and org charts demonstrate culture vividly.
    – Stuart Ainsworth
    Commented Nov 9, 2017 at 15:13
  • @StuartAinsworth indeed, that's why I didn't talk about modifying the organization but more to on-board the rest of the team.
    – Tensibai
    Commented Nov 10, 2017 at 9:40
18

The most important thing for DevOps Engineers in this kind of situations, is to get (a) Management Commitment and (b) Required Budgets. Read on for some more details on both ...

Get Management Commitment

Once that is in place, things become easy for such DevOps engineers. Especially whenever resistance (from all sorts of parties) comes into the game. Trust me, there will be such resistance, which challenges such as:

  • Why do we have to change? I want to keep doing what I did for X years already!
  • I do not want to loose the (technical) power I have, and complete all sorts of workflow procedures, to get a silly fix in production that should take me like 5 minutes instead of 5 hours (or days ...).
  • ... (I could add another dozen bullets here ...).

Whenever those challenges come up, all a DevOps engineer should have to say is like:

I'm sorry, I'm just doing my job ... based on instructions from upper management.

Get Required Budgets

An effective way to get Required Budgets, is to create/submit an appropriate business case which explains the tangible and intangible benefits of various DevOps practises by applying them to some real world cases that apply to the company itself.

Below are some real world cases I experienced myself, as an SCM consultant hired by some companies where these things had happened. I know, SCM is only part of DevOps, but it's the area where I have some expertise ...

1. Benefits of automation

Due to some strike from only 2 (!!!) computer operators (who did not type the console commands anymore that they where expected to type), trains had to be halted somewhere half the way between 2 factories (since the system at the factory where the train was heading to was down, crucial data about handling the train was not available).

By implementing an SCM system, many operator commands were automated.

2. Reduce software license costs

Some software vendor had decided to increase some yearly fees for the (outdated) SCM software, which management didn't agree to. Therefor they created a special project to get it replaced by some alternative SCM software.

The budget of the project was equal to the yearly fee they didn't want to keep paying. That included flying in engineers from other continents (like me) to make the project succeed.

3. Reduce operating costs

Some major insurance company was using some FTP software to transfer software fixes to about 13.000 midrange computers (AS/400s) throughout the country, and this whenever "a" fix became available. The cost of 1 such transfer was about 4 USD (13.000 x 4 = 52.000 USD for a single transfer ...). The software consisted of 120.000 components, developed/maintained by about 150 developers. Do the math about the probability that 1 developer made 1 (tiny) mistake in any of these 120.000 components, which made it to production, and required an urgent fix, which would cost another 52.000 USD (just for the transfer!).

By implementing an adequate SCM system (with managed test environments, approvals, etc), this company achieved a major cost reduction. Think about it, if the SCM system could prevent the need for only 20 transfers of urgent fixes, it resulted in a cost reduction of 52.000 x 20 = 1.040.000 USD (quite a budget to implement an SCM system, they only needed a fraction of that amount to get the job done).

4. Reduce costs of unavailability

If the above cases are not convincing enough, then think of the system(s) of a major credit card company being unavailable around the world. I've been told that 1 second of unavailability costs them 1.000.000 USD.

That's probably also the reason why, for a very long time, such companies have sophisticated DevOps tools in place, for many decades already. Because each second they are not in business costs them a fortune.

Improve this answer
1
  • I think you're missing some steps. If the dev team isn't already deploying multiple copies of the application (at least a test environment before prod), then that should be the mandate. Then they can maybe struggle with that on their own for a while if they really want to spend the time. This makes a Dev Ops expert helpful to these people; they can turn a painful process into a smoother one, instead of resorting to "management says so." That's where the entire idea of Dev Ops comes from, after all: eliminating the pain of deploying and managing multiple environments.
    – jpmc26
    Commented Nov 9, 2017 at 22:49
5

TL;DR: Since upper management is usually fickle and prone to anger, I'd suggest trying to bend his mind a bit to get a different perspective, while changing things for the better, gradually.

(I am assuming that his trouble is mainly with the reluctant devs, not his other ops colleagues who seem to do classical operations mainly.)

IMO, even if you have DevOps going, that does not necessarily mean that every developer needs to be a full fledged DevOps guru. I find it rather normal that there will be one or two real experts in a given group of people, and the rest more or less tag along. As long as the workload is not too big for that guy, and as long as he manages to encapsulate his knowledge in scripts etc. instead of building his own silo, it is fine with me.

The one thing that should not be happening is that the DevOps guy does his automation, and everybody else tries their hardest to avoid said automation (i.e., go past the CI/CD pipeline and do stuff manually in one of the environments). This, IMO is the main thing that has to stop. One solution for that would be to push very hard for the cattle-not-pet approach, i.e. relentlessly tear down VMs or containers left and right as soon you can, and spin up new ones continuously.

Second, of course everybody needs to be aware of what the automation is doing, and at least in theory, with some digging around maybe, be able to start the automation machinery (i.e., if everything runs from a commit/push, then the devs need to be aware and very much up to date with the fact that there will be stuff happening in the background when they commit). The CI(/CD) pipeline needs to be quite visible and should be a thing everybody is constantly aware of (i.e. when a dev breaks it).

Third, the "one guy" of course has to watch out that he does not do menial, everyday tasks for his colleagues (e.g., creating Dockerfile after Dockerfile for their artifacts...).

Fourthly, the solutions the DevOps guy creates of course have to actually be superior to the manual approaches of the past in some measurable way. In that case, it should be possible for him to demonstrate his improvements; i.e., demonstrate how things get easier for everybody, or how it seems to be impossible to introduce bugs in the later stages of the pipe etc. If this does not seem possible, then the DevOps guy needs to take a good long hard look at what he is doing. If it is possible, that calls for brownbag sessions and plenty of evangelisation in his team.

Obviously, in such a reluctant environment, you will probably not be arriving at a wholly automated CD solution, or at trunk based development anytime soon. But I would not worry too much about being singled out. He is the expert, and if he is doing his job well, the whole team will very gradually improve.

And finally, if after years and years of toiling there is just no visible improvement with his colleagues, it is always possible to look for other avenues (be it inside or outside the company). Having all that DevOps experience under his belt is excellent base for looking for a job, these days...

Improve this answer
5

I see a lot of great answers here talking about DevOps as a culture, suggestions about how to work with management, and help defining the do’s-and-do-nots of a DevOps team or engineer. I think each of them is great, and really illustrative of many answer can be 100% right, and still be very different, or even completely ambiguous, from each other... That’s DevOps!

This answer is just my unique perspective from experience, and may not be indicative of norms or best practice...

But what your DevOps colleague was complaining about is the very nature of what makes DevOps challenging and difficult, especially when appointed the role of DevOps engineer, and not simply being a cultural mindset.

Personally, I like being a lone wolf, because I still make valuable contributions, but also get to set my own limits, whilst persuading others to help themselves, thereby helping me, breakdown IT silos.

Some silos remain intact, and that’s okay, it’s DevOps mission to work around that and try to make that silos as insignificant or invisible as possible.

It’s possible your coworker may be realizing, or hasn’t yet realized, that he or she does not like being a DevOps engineer.

Improve this answer
3

Relatively speaking, the concept of devops is new and still defining itself in my opinion. I currently fulfill a devops engineer role. For me, this means I facilitate and develop the tools and processes used by both our dev and ops teams freeing them to focus on the product that generates revenue for the company. The ops and dev teams spin up their own servers and such as needed. I just hook up the CI for our products, ensure our processes makes sense and seek out what process can be improved/automated. I meet with all of our departments, from sales, to warehouse, to developers and operations (QA and release managers) to see what they are doing and how I can improve their process.

Improve this answer
2

For me, DevOps means that the development and operation of a software system becomes the responsibility of one team, instead of separate dev and ops teams. This is a two-way street. The best teams are made up of "T-Shaped" people who are experts in one field and familiar with several related fields.

  • Team members with Ops experience are expected to do what they do best (i.e. Ops), to teach the fundamentals of what they do best (i.e. Ops) to the others, and to learn and do jobs in related fields (i.e. Dev tasks).
  • Team members with Dev experience are expected to do what they do best (i.e. Dev), to teach the fundamentals of what they do best (i.e. Dev) to the others, and to learn and do jobs in related fields (i.e. Ops tasks).

So to make the DevOps engineer feel less like a lone wolf, invite him to teach the developers how to run the systems, while acknowledging that he is the expert how to design the infrastructure.

Get him involed in high-level architecture from the beginning, so that he can introduce the concerns of his specialty. (Before we had DevOps, our architecture drawings always glossed over "little things" like load balancers and redundant servers. Now such things are part of the very first sketches.)

Expect the Devs to take some of the daily routine Ops tasks, both to build redundancy into the team and to spread the "drudge" jobs fairly.

Expect him to contribute to the Dev effort if there are no Ops-like tasks to be done. Some DevOps I know seem to find database work a natural extension of their area of expertise, not sure if that can be generalized.

Improve this answer
1

What could be done to help DevOps Engineers feel less like a lone wolf?

To paraphrase - what can a DevOps Engineer do himself/herself to feel less like a lone wolf?

Lack of culture and management support is only one part of the equation. The other part is in my opinion that the detail DevOps knowledge often refers to complex contexts and it is important to have advice and reference to working examples.

Therefore - do not feel like a lone wolf; participate in DevOps communities like this one here or tool-specific groups and GitHub - the feeling is at least then you are not the only lone wolf ;-)

Improve this answer
1
  • 2
    DevOps, by it's nature is a team exercise. The only thing a DevOps engineer can do by him or herself to feel less like a lone wolf, is to quit and go join a more functional organization.
    – James Shewey
    Commented Nov 13, 2017 at 16:28

Not the answer you're looking for? Browse other questions tagged or ask your own question.