One or multiple git repositories for independent small parts of one larger project?

Question

We are developing an Azure-based analytics platform that consists of several modules. Among other things, Azure functions, scripts for creating the infrastructure and scripts for initialising the database structure including the creation of stored procedures are used.

Currently, the repository structure is a bit messy and I was asked to clean it up and develop a strategy for deploying and updating individual parts of the platform (We currently redeploy the entire platform with every release).

What is the best way to implement this at repository level? For example, do I create a separate repository for each Azure function and the associated pipelines, or do I put them all in a common repository and try to solve this via individual release branches for each Azure function?

Any other best-practice approaches?

Answer 1

This is something I also struggle with. I can tell you from experience it depends on the type of application and the tech stack being used. The short easy answer is that having everything in one repo is usually not the wrong or hard way. Breaking things out often doesn't offer many benefits but again it will all depend on your CICD engine and what kind of application and what tech is being used.

Here recently I had a project that had 3 repos. 1 for a CLI application. 1 for the infrastructure as code being deployed by the CLI and 1 for a library that was used by the CLI application and other tools.

This all seems straight forward and a good choice but the CICD has been really pain full.

Often a new feature will span across 3 repos. Trying to get this to work in CICD is impossible as far as I can tell. I would much prefer to have the infra and CLI together as most new features affect both codebases.

Then I could at least get a list of the changed files in the CICD pipeline and decide if I need to test the infra or not.

The other issue when you split things up like this is trying to version these different parts and understand what versions work with other versions etc.

I guess to sum it up would be if things depend on each other then keep them in the same repo as it really helps the CICD process. Think an application and its underlying infra. Now would I combine a rest api and its client? probably not, in the past I just required the branches be the same name. So in CICD I would check if the other repo has a branch of X and if so use it and if not use master. If things are mostly independent like a library and a application that uses it, then keep it split up.

Answer 2

There's no easy answer to this, get your repos too big and every branch becomes huge over time, make it too small and you end up doing a lot of administration.

Many smaller repos has my preference, but only after creating some support scripts to do the administration for you.

For example, to pull all repos:

ls -d **/ | xargs -P12 -I{} git -C {} pull

Also git has a feature built in for repos, submodules.

https://git-scm.com/book/nl/v2/Git-Tools-Submodules

They come in handy when a repo references another but does not to do source control on the other. It comes with its own subset of commands. It's worth looking into this