Tips for separating DevOps Git repos?

Question

We currently have a Git repo with automation code (with tests), iaas-code and lamba-code.

• The automation code is mostly Fabric, Python and unittests.
• The Iaas-code is mostly CloudFormation, Docker-compose and some Terraform.
• The AWS lamdas are Serverless code and some Fabric.
• There is also some code with Dockerfiles that builds a Docker image from this Git repo.

We're considering separating git repos and using git submodules as it seems much cleaner. Do you have any tips for this process?

My concern is that most tips regarding mono-repos vs. separate repos are mostly for programmers that build micro-services and not for DevOps / SysAdmins.

Answer 1

When considering mono-repo vs many-repos you might look at the following questions, which should also affect the composition of the code directory tree structure. There is a slight general trend toward mono-repos.

Discoverability

How easy it is for developers and ops people to discover the code for some feature. Are there clear guidelines on what goes where? Are they enforced somehow through automation? Are new repos automatically linked as submodules into correct location? Does the build or deploy process depend on the tree structure making sure people need to put it there in order for it to be even noticed? What is the size? How long it will take to grep code in certain place? Are there generic lib/ directories where devs and ops end up sharing libraries that create new unexpected inexplicit dependencies?

Deployability

Does any of the code need to be present on machines that you deploy to? Are you expecting to do such deployment through syncing repository or do you have other established deployment mechanisms. In the first case, will other code creep in during the deployments that does not need to be there?

Durability

Will the concept you are using stand the test of time? Are you a one product shop, but maybe in your future there are new products? How would adding a new product affect the code? Will there be new mono-repo for the new product? What about shared code between products? Looking at your company roadmap, how will whatever you do fare 6m, 2y, 5y down the road? Choices like this tend to be hard to change in practice (even though they shouldn't - in theory).

DevOps

You really want to take down barriers between Devs and Ops. So even just having separate mono-repo for ops could create such barrier. The operation and deployment of code, standing up environments, setting up services, it should all be in reach of developers and with explicit expectations to develop such code in coordination with the ops team, not being dependent on someone doing it for them. If things go well your developers follow through with code for months after it has already been in production and have hand in all aspects of the deployment. How easy it is to make commits that contain both product and deployment changes? How can those commits be grouped together? What is the visibility of ops into product runtime code and devs into product management code?

Answer 2

My advice would be to not separate these repositories as it introduces unnecessary risks in terms of repositories not being available due to access control, availabilty or billing issues at some point when you need 2 or more of them working together.

It also reads contrary to the term "DevOps" where Software developers and System admins are supposed to work together and get a grasp of each others domain instead of traditional "over the wall" responsibilities. If Dev and Ops are supposed to work together, why would Ops and Ops be separated?

Answer 3

I lean towards more repositories or submodules.

Sure there might be some more configuration to do (hooks, permissions etc.), but understanding a repository with 5 files in it is a lot easier than understanding one with 5 folders.

As you mentioned, programmers use different repositories when implementing micro services. They split them up logically based on interfaces or APIs. As a sys admin you could split up repositories by defining logically what each of your components are responsible for.

For example, if your automation code is for testing/monitoring your Iaas I would put those together in the same repository. The tests verify the validity of your infrastructure. As for your lambda functions, could they potentially be reused for other types of Iaas? Can they be extended for other functions?

Also ask yourself if someone new started today, could they understand your repo and submit a pull request? Or would they be overwhelmed?

Less complex repositories enable other users to be able to make meaningful contributions in less time.