How to achieve a smooth transition from “the one big VCS repository for all products” organisation model to the “many small VCS repositories” model?

Question

It is a common scenario that the codebase of a product held by a repository in some VCS system evolves to a point where that codebase can arguably be seen as containing several products. Splitting the codebase across several VCS repositories, each dedicated to a single product, can leverage several benefits (see Benefits of having a product per VCS repository over the bloat repository model below). On the technical side, splitting the codebase is a rather easy step as most VCS support this operation. The split however might rise engineering issues related to automated testing, continuous delivery, service integration or monitoring (see Issues raised by the split.) Organisations planning to perform such a split therefore need to know how to perform this transition as smoothly as possible, that is, without interrupting their delivery and monitoring pipeline. The first step of this is probably to better understand the notion of project and how to delineate the split in a monolithic codebase.

In the answers to this questions, I would like to see:

1. An attempt to give a working definition of what a product is, which gives practical criterions to actually delineate products in an existing codebase.

2. According to this working definition, elaborate a plan that actually perform the split. We can make the simplifying assumption that the codebase is processed by a fully automated sdlc implementing continuous-integration and continuous-delivery. That is, each branch is validated by an automated testsuite implemented in the current codebase and each merge to some “magic” branch generate product-artefacts that are tested and deployed. (Product artefacts are e.g. source tarballs, documentation, binary software packages, Docker images, AMIs, unikernels.)

3. Such a plan is satisfying if it explains how to circumvent the

Issues raised by the split

1. How automated testing procedures relate to the pre-existing monolithic repository and the split repositories?

2. How automated deployment procedures relate to the pre-existing monolithic repository and the split repositories?

3. Where is stored the code for automated deployment procedures themselves?

4. Where are stored infrastructure, monitoring and high-availability strategies?

5. How to ensure that a developer needs only one codebase at a time (but possible uses artefacts from other codebases).

6. How can a tool like git-bisect

---

Marginal note: Benefits of having a product per VCS repository over the bloat repository model

Having several small repositories holding the codebase for a specific product has the following advantages over the “bloat repository” approach:

1. With a bloat repository it is hard to roll back a release when a product is unstable, because history is mixed with other product history.

2. With a bloat repository, it is hard to review project history or pulls, with small repositories, we are more likely to read this information. (This might be specific to VCS like git, where unlike svn, we cannot checkout subtrees!)

3. With a bloat repository, we have to do much more branch-dance when we develop. If we have N repositories we can work in parallel on N branches, if we have only 1 repository we only can work on one branch, or have a load of working copies which also are a hassle to handle.

4. With several small repositories, the logs give a heat map of the project. They can even be used as a proxy of knowledge diffusion in the dev team: if I did not commit in repo X since 3 months, it could be good to assign me in a team working on repo X so that I stay aware of the developments in that component.

5. With small repositories, it is easier to gain a clear overview of a component. If everything goes in a single big repository, there is no tangible artefact delineating each component, and the codebase can easily drift towards the big ball of mud.

6. Small repositories force us to work on interfaces between components. But since we want to have a good capsulation, this is a work we should do anyway, so I would count this as an advantage for small repositories.

7. With several small repositories, it is easier to have several product owners.

8. With several small repositories, it is easier to have simple code standards that are pertinent to a full repository and that can be automatically verified.

Answer 1

It is a fascinating question for which real answers may not actually exist; I appreciate that while you tried to keep the question contextualized on the VCS, it naturally scaled by itself up to infrastructure design and implementation planning.

Though, it seems many of us are working of this kind of transitioning, which can be exciting, but at the same time so frustrating and painful; and I would like to share my experiences and views, trying not to be pedantic, and just because I may not be such a good engineer, also not to be boring.

Design

Infrastructure and architecture should go together to write a modern software. Tightly coupled, if you want. It might sounds weird to use those words, but we are not certainly talking about code itself here: I mean they must be part of the same blueprint. When the clouds arrived, and people started to write software for them, how many people then realized that by putting the mudballs there, they just would be the same mudballs in a different place(?) Maybe a few forward thinking people could foresee that, and they are likely working in devops today. As devops is just a buzzword with so many differnt meanings for differnt people, I have seen places in which the devops team would sit in every architecture meeting; other places in which is automation only. To achieve this kind of transformation, we have to fight our way to sit there.

Confidence

The transition must be kept isolated, in the sense that a consistent cut of history must exist, that provides the transition itself and itself only, without any other change (after several months of preparation). With what confidence one would approve it and push the red button?

I mean the codebase must change to accommodate the new VCS structure, and it will be very difficult to keep it merged during development. (for this issue there may be facilitating strategies, I'll talk about a common one later, that can help parallelize development a bit).

Well I bet the only way is with behavioural testing, and the same suite of behaviour tests should be launched to verify the old with new codebase. We are not verifying the application behaves as expected here, but that the transition does not alter the behaviour. Having failing tests may be a good thing! If they continue to fail!

In fact it is very uncommon for mudballs to be well tested; usually the code is very tightly coupled, and likely, for most legacy code, it was not developed with a test driven approach, not even unit tests.

If such test code is missing whatsoever, it shall be written first.

Strategy

Yes, the transition must be kept isolated; but at the same time integrated. I know I may sound crazy here, but I wouldn't find other words to describe how confidence can keep up to business. Very few, if none at all, companies would like to stop development for a big monolithic codebase, to make space for this transition, and we are not making it just happen within a toss of a coin. Maybe hundreds of developers might be continuously contributing to the codebase (I would use the tampering word here, from our POV). While our work must address a specific snapshot to provide confidence, we have to keep ourselves rebased (not in a git meaning here), to avoid to fall behind forever.

The implementation strategy in here can give different experiences. A common line of development is to wrap/adapt (exposing endpoints with optionally rearranged schemes) newer implementation branches (well, living in other repositories in this case), when they need to interact with the core. Transitioning with a strategy like this, along with refactoring, can at the same time offer a POC scenario for the VCS transition, and later on a step by step approach. See it like sculpting the ball of mud. Yeah life offers so many funnier things.

Technical Debt

Business management spheres started to understand technical debt and keep it into consideration. Nope, scratch that, not true. While is it increasingly common to gather measurements and quality data, in terms of static code analysis, code reviewing, behavioural test results, and performance, and generate nice reports and everything... it still remains incredibly difficult to make the business accept a continuous refactoring approach. The business value of it. "We are following an agile process, and this will not bring in any enhancement to the features, wouldn't it?". Basically, by doing so they are negating the existence of technical debt. I see this as the common missing necessary step to be able to start any transition from monolith to microservices architectures.

Reaggregation

After all this, you might still want to provide a single repository-like view in which you can build more than one single product. For any reason, ie curr/next release, multibrand, customer builds. Tools like google repo may help in this case. Never used one myself, but I know I'll need one day.

Integration testing

With microservices, integration testing assumes a different context, of "testing own API". Higher layers of testing, functional or performance, can and will exists, but do they mean much without proper integration testing? It would be like having integration testing without unit testing. Of course not. That's why, if you need to git bisect, you will execute it in a microservice repository branch, forget about running that in the mudball repository.

PS. this is a draft, my English is bad and I'll fix it one day