What are the signs of an understaffed DevOps team?

Question

What are the typical signs and signals of a DevOps team being understaffed? How would you justify/explain a request for a new addition to a team?

---

I would love to keep the question generic, but here is some additional information:

We currently have 2 DevOps specialists working together as a team, but the demands and the amount and complexity of products are growing. We are thinking to request a new addition to the team, but having some difficulties explaining and proving why it would be a good idea.

Answer 1

There are four main reasons why you can feel your team is understaffed:

1. Poor organization and planning of work
2. Doing work someone else should be doing
3. Doing work that should not be done at all
4. Being actually understaffed

Start with a review of the first three points. Read the Phoenix Project on ideas how to do the first. Ask yourself for every task you help anyone with if it should be done at all and if it is you that should be doing the task or if you should simply enable whoever needs it done to do it themselves. This will give you some documentation on why all the work you do is necessary.

Next review the four types of work mentioned in the Phoenix project:

1. Business Projects - what you do for other teams in the organization
2. Internal Projects - what you do to make your work easier in future
3. Scheduled Maintenance - what you do to keep the lights on
4. Unplanned Interrupts - what you do because something went wrong

If the work of your team is sustainable, you will spend roughly the same amount of time on each of the four. If the unplanned work starts to creep up close to 50% of your time, it is a sign you are definitely understaffed.

You should be able to hire to stay about one person ahead of the unplanned work reaching 25% of your time, otherwise, one person leaving will send your entire team into a tailspin you might never recover from. Overprovisioning of people and technology have same reasons and benefits.

Answer 2

Background: Besides for providing support to our current infrastructure and to our Developers, we do monthly planning as a DevOps team for what we want to accomplish on top of helping dev teams within sprints and new projects that are launched. However, during the month we often notice extra things that need to be done and improved, which we then add to our backlog. We also are responsible and assist with various other things that fall beyond our scope, but we assist the business were we can :)

Answer: As soon as you notice you are not getting round to or postponing lots of tasks especially maintenance, I think that is a good indicator(from what I have experienced). Also, the more new projects and dev teams that come in the thinner the DevOps team gets spread, the more people you will need.

Its super easy just to get caught up in the day to day completing tasks, but I believe its super important (even once a month) to take a step back and assess this.

Answer 3

I actually take a page from the SRE Handbook on this one, which I think is very relevant. DevOps specialties are not meant to grow horizontally with an organization. Rather, if you see that things aren't getting done then it's a signal you're not properly empowering developers to self-service.

Evaluate your processes and see how they align to the commonly accepted DevOps Principles and how well you're following industry best-practices.

Answer 4

I assume this team of two is going from project to project and establishing DevOps stuff there (creating CI/CD pipelines, supporting the other devs creating Dockerfiles, or whatever technology you are using). In other words, type 3, 4, 5 or 6 as per http://web.devopstopologies.com/ .

In this case, a sign of shortage is simply too much workload for those two; too many projects requesting their services; too many tickets; overtime; stress, burnout. These factors should be reasons enough for a responsible leadership to add more capacity. I don't see a DevOps specific sign in this, it's just a function that is understaffed.

Another sign to change something is if you take a good hard look and if you notice that you are creating a "DevOps silo", in which all DevOps know-how is concentrated in those two guys/gals, and everybody else just leans back because those two are "doing DevOps". That is not the point of DevOps. If this is the case, think about the cultural aspect, and modify them to be more evangelists/teachers/coaches for the other teams.

In both cases, the deeper reason of why having DevOps in the first place is a good thing (the general Good Stuff) should be clear to the upper management. If you cannot bring that message across, then scale down the work which your team is doing, by shifting it onto the regular Devs/Ops (as should be the case, anyways).

Answer 5

I was under the impression DevSecOps was a mindset, not a team - if you have a Dev(Sec)Ops "team" you're doing it wrong... I'm trying to wrap my head around putting two "DevOps Engineers" together and calling them a "DevOps Team."

We have development teams, SCM, Application Security and Systems Engineers all working in tandem for a rapid deployment/release model for pushing code and configuration/system changes through to a given end point - either staging or production

This has nothing to do with any "devOps" engineers, as such.

Answer 6

Grouping of tasks

An approach we've used in the past in similar situations is to organize the work of a team in 4 major groups of tasks, and allocate the equivalent of 2 FTE (Full Time Equivalents) to (try to) complete those tasks. In our case it was related to running an SCM helpdesk in a mainframe environment, with about 300 developers asking for all sorts of help / interventions from those 2 FTEs. The groups of tasks are organized in 4 possible priorities:

• Tasks of Priority 1 and 2 must be completed (no excuses / negotiations possible)
• Priority 3 tasks are to be completed "as soon as time permits".
• Priority 4 tasks are to be completed "if time permits".

Read on for more details about the kind of tasks in each of those 4 groups ...

Task descriptions

Priority 1 - Operate the helpdesk

• By experts who are easily accessible and always available.
• Via phone, eMail or ticketing system during business hours.
• Compliant to predefined SLAs.
• ITIL-based registration of all helpdesk calls with periodic reporting of all calls.
• Apply emergency customizations (work arounds) for critical issues.

Priority 2 - Watch duty services

• 24h/day, 7d/week on-call availability.
• Compliant to predefined SLAs.
• Reporting of all watch duty calls.
• Management escalation where needed.

Priority 3 - Routine maintenance

• Administration.
• Application boarding.
• Housekeeping.
• Performance enhancements.
• Space management.
• Tuning of resource consumption.
• Suggest enhancements for customisations to reduce the number of helpdesk calls and/or watch duty interventions.

Priority 4 - Fixes and enhancements

• Create and maintain user documentation.
• QA testing of new customisations.
• Develop and implement enhancement requests.
• Participate in DRP-testing.

Evaluation

If you're using an approach as described above, various things may (will!) start to happen:

• If the team is understaffed, probably most of the time will go to priority 1 and 2 tasks, while it may take a while to get the priority 3 tasks ... and priority 4 tasks might suffer starvation (there appears to never be time for those tasks).
• The more there is (becomes) time available to "invest" in priority 4 tasks, the more time needed for priority 1 and 2 tasks will be reduced, so that even more time (of the available budget of 2 FTEs) can be "invested" in priority 4 tasks.
• You'll be amazed to see how, after a while, the number of priority 1 and 2 tasks will go down. And if you do adequate reporting about those tasks, that's something management loves to hear. In our case that number went down from about 300/month, to below 100/month ...
• If however the 2 FTEs appear to never (or hardly) have time for priority 4 tasks, then you have a perfect explanation and proof for your management ... that you're understaffed.