How can continuous integration scale for very large projects/teams?

Question

Traditionally CI systems only monitor the quality of the code in an integration branch, signalling when regressions occur. Human intervention is required for repairs.

As the number of developers working on the same branch increases the risk of breakages/blockages increases. Eventually a point is reached where on average by the time a breakage is fixed a new one appears—progress on the branch becoming effectively negligible.

Splitting into multiple teams each working on a separate integration branch which would be merged at a later moment might help, but greatly extends the project duration, since it simply delays the necessary integration for a later moment while adding churn/noise/technical dept related to the partial integration from individual branch merges. Costs also increase due to CI setups for each of the branches, each with their own build/QA resources, etc. And overall quality is not necessarily better.

Scalability is, at best, dubious.

Is there a method for making such large projects actually scale?

Answer 1

Yes, but it's by preventing the things you mention in your question as much as possible. You are right that there are only so much developers that can work on the same code before things become unmanageable. You need someone or something that has an overview of all changes and makes sure integration works fine and regression doesn't occur. This is hard to scale if everything happens in one repository.

Before starting any work you need to separate functionality in such a way that things can be integrated later with little or no effort. This means that you want to split functionality into parts that are independent, or at least have as little dependencies as possible. How you do this depends very much on the product you are developing. In the past separation of functionality was done by creating different libraries. The downside to this is that you need a good method of managing those libraries. Old versions of Windows for example had a bad library management system which caused the infamous DLL Hell. Nowadays micro-services is (becoming) a popular way to do this, although this also has it's drawbacks (e.g. communication overhead, more services to monitor)

If separation based on functionality is difficult, another option may be to separate development in time. Instead of multiple teams working simultaneously on many different features, you have 1 team that works on only 1 or a few features at a time.

When separation of work is not possible, you'll need tools and tests that make sure things align. Automated unit and integration testing can be a big help here, but ultimately there will be limits to how big a project or team is to be workable.

Answer 2

As the number of developers working on the same branch increases the risk of breakages/blockages increases. Eventually a point is reached where on average by the time a breakage is fixed a new one appears

While the first part of the sentence is probably statistically correct I disagree with the second.

CI and integration branches cannot defeat GIGO (garbage in garbage out) If the developers has no discipline then you will end up in a halted branch, but integration branch tests should only act as the lat barrier, not the first.

Developers should use proper coding standards, code reviews, local unit tests and possibly pre-integration branch tests to make collisions less likely.

I am working daily on such a system with hundreds of contributors and surprisingly it's uncommon to see it hangs.

I don't have enough data but it might be true that cost is high, on the other hand it's difficult to judge the alternative cost of late integration and bugs.

Answer 3

Another approach is to switch to a different, non-traditional CI system, capable of preventing regressions by identifying and rejecting the offending candidate changes prior to their commit rather than detecting regressions and waiting for recovery while development on the branch is blocked. I like to call such system a non-blocking CI system.

Simply bumping up the developer-executed pre-commit verification requirements in an attempt to reduce branch regressions is not sufficient for such guarantee. It may actually cause a regression rate increase: longer verification times => higher number of changes simultaneously verified in isolation (thus not accounting for each-other) => higher risk of functional conflicts and interference. I'm explaining this in a bit more details in the Extending the pre-commit verification coverage should increase our branch stability myth.

The CI tool also needs to be capable of operating at scale, of course. And such tools exist, see Is there a CI tool that guarantees no regressions in the branch quality level?

^{Disclosure: I'm affiliated to the company mentioned in the above-links.}