8

Traditionally CI systems only monitor the quality of the code in an integration branch, signalling when regressions occur. Human intervention is required for repairs.

As the number of developers working on the same branch increases the risk of breakages/blockages increases. Eventually a point is reached where on average by the time a breakage is fixed a new one appears—progress on the branch becoming effectively negligible.

Splitting into multiple teams each working on a separate integration branch which would be merged at a later moment might help, but greatly extends the project duration, since it simply delays the necessary integration for a later moment while adding churn/noise/technical dept related to the partial integration from individual branch merges. Costs also increase due to CI setups for each of the branches, each with their own build/QA resources, etc. And overall quality is not necessarily better.

Scalability is, at best, dubious.

Is there a method for making such large projects actually scale?

  • 1
    This question is, to me, like quite a few other questions that make me think like "why is this even a question". Not that it is a bad question, but rather something that in the world I'm coming from (mainframes, I bet you know by now ...) is not really a question anymore. That's what over there we're doing for at least a decade or so already, and IMO it works like a charm. How otherwise would it be possible to keep a banking system going. I just don't know if it makes sense to post an answer looking at it with mainframe glasses. What do you think? – Pierre.Vriens Mar 10 '17 at 16:06
  • 1
    The scale of the project here means the size and speed of the changes to the system being performed, not necessarily the size of the system itself (which would matter, for example, if the project would be re-writing the whole system or building it from scratch). And the context is a single branch, detaching on feature branches for days/weeks/months at a time is not continuous integration. Not saying that it doesn't happen at your place, only saying that the mainframe context itself doesn't imply large scale continuous integration. Think hundreds of commits per day into the same branch. – Dan Cornilescu Mar 10 '17 at 16:32
  • This question leaves me with so many more questions - how did you end up in this situation? Why are so many developers working on the same branch? Why would feature branches need their own QA environment? Why are changes being pushed to the integration branch that break the build, and why isn't that the problem being addressed rather than CI scaling? Are they not testing locally? Not merging from upstream before they merge to it? – Adrian Apr 28 '17 at 15:36
  • 1
    @Adrian Branches are evil - they bring the Integration Hell with them - risky, slow, expensive. Especially for large teams. Branches can easily diverge too much and make merging impractical. Lookup the CI advantages. Branching means working in isolation - canceling all those advantages and bringing back the complementary disadvantages. Yes, branches have been used to successfully ship sw for decades - there was no alternative. But now we have CI. Yet many still use branches - they're used to them, not even feeling the pain anymore or choosing to ignore it. Or hit scalability issues. – Dan Cornilescu Apr 29 '17 at 5:26
  • CI solves a completely different problem than branches, and used correctly, branches work with CI, not against it. Not using branches doesn't avoid integration hell either, it makes it worse: everyone working in the same branch has to merge every time they push. – Adrian Apr 29 '17 at 14:55
5

Yes, but it's by preventing the things you mention in your question as much as possible. You are right that there are only so much developers that can work on the same code before things become unmanageable. You need someone or something that has an overview of all changes and makes sure integration works fine and regression doesn't occur. This is hard to scale if everything happens in one repository.

Before starting any work you need to separate functionality in such a way that things can be integrated later with little or no effort. This means that you want to split functionality into parts that are independent, or at least have as little dependencies as possible. How you do this depends very much on the product you are developing. In the past separation of functionality was done by creating different libraries. The downside to this is that you need a good method of managing those libraries. Old versions of Windows for example had a bad library management system which caused the infamous DLL Hell. Nowadays micro-services is (becoming) a popular way to do this, although this also has it's drawbacks (e.g. communication overhead, more services to monitor)

If separation based on functionality is difficult, another option may be to separate development in time. Instead of multiple teams working simultaneously on many different features, you have 1 team that works on only 1 or a few features at a time.

When separation of work is not possible, you'll need tools and tests that make sure things align. Automated unit and integration testing can be a big help here, but ultimately there will be limits to how big a project or team is to be workable.

4

As the number of developers working on the same branch increases the risk of breakages/blockages increases. Eventually a point is reached where on average by the time a breakage is fixed a new one appears

While the first part of the sentence is probably statistically correct I disagree with the second.

CI and integration branches cannot defeat GIGO (garbage in garbage out) If the developers has no discipline then you will end up in a halted branch, but integration branch tests should only act as the lat barrier, not the first.

Developers should use proper coding standards, code reviews, local unit tests and possibly pre-integration branch tests to make collisions less likely.

I am working daily on such a system with hundreds of contributors and surprisingly it's uncommon to see it hangs.

I don't have enough data but it might be true that cost is high, on the other hand it's difficult to judge the alternative cost of late integration and bugs.

0

Another approach is to switch to a different, non-traditional CI system, capable of preventing regressions by identifying and rejecting the offending candidate changes prior to their commit rather than detecting regressions and waiting for recovery while development on the branch is blocked. I like to call such system a non-blocking CI system.

Simply bumping up the developer-executed pre-commit verification requirements in an attempt to reduce branch regressions is not sufficient for such guarantee. It may actually cause a regression rate increase: longer verification times => higher number of changes simultaneously verified in isolation (thus not accounting for each-other) => higher risk of functional conflicts and interference. I'm explaining this in a bit more details in the Extending the pre-commit verification coverage should increase our branch stability myth.

The CI tool also needs to be capable of operating at scale, of course. And such tools exist, see Is there a CI tool that guarantees no regressions in the branch quality level?

Disclosure: I'm affiliated to the company mentioned in the above-links.

  • You mind if I rework your "disclaimer" a bit (if you don't like it, just perform a rollback)? – Pierre.Vriens Mar 6 '17 at 23:32
  • Not at all, please do. – Dan Cornilescu Mar 6 '17 at 23:51
  • 1
    Done ... IMO equivalent to your version, compliant to SE requirements (to add a disclosure), and a more appropriate (less disturbing) font. Feel free to rework or rollback if you want to. – Pierre.Vriens Mar 7 '17 at 0:15

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.