Pooled Build Server and Multibranch development to stabilize the trunk?

Question

I'm a Dev in a big (40+ Devs) old (16+Years) SW project. We still work with SVN as source control but we are getting after automation as much as we can.

One problem that hounts us is, that 40 devs check in stuff into the trunk without a fair chance of running all the tests. Our test suites consists of JUnit unit tests, Junit integration tests that run inside our app server and test business processes and some test tool blackbox tests that run after the deployment. Unittests run 10Min, Integration tests 40Mins and test tool tests run all night long.

You can imagine that for each check in, not ALL the tests and most of the time also not all of the Integration tests are run. Resulting in regular test failures on our CI environment (thats the good part: we have one!) But with 40 devs there is to much going on at the same time, so nobody knows for sure why a test failed. "Was it me? I changed something at another part of the app. This could not be the case, right...?"

I try to promote a branch based dev process with short(!) lived branches that could be testet before merging them into the trunk. Switching to Git is part of it but I would not be religious about it. Goal is that the trunk is always green. But for this I need an infrastructure that can support up to 60 branches.

I'm dreaming up a pooled build server system, where each team could request that a certain branch (maybe in a given revision/checkIn-ID) should be build and tested (maybe with a given testsuite). All the build requests get into a queue and are processed by the build server fleet. (Currently we have the problem that not enough hardware is available to scale the build infrastructure. Pooling the available hardware seems to be the best approach in my mind.)

What do you think about this approach? How would you do this? Is https://jenkins.io/doc/book/pipeline/multibranch/ the right path down the road?

Am I going down the wrong path here? I know that CI says that everybody checks in into trunk at least every day. But, damn, there are so many side effects when 40 devs check in together and the software (especially the data model) does not prevent side effects between parts of the application. We have to start somewhere. But where?

Answer 1

Attempting to run your validation on branches, before being integrated into the trunk won't get you very far. The reason is that fundamentally such validations are performed in isolation, most often they will not actually reflect the trunk picture after the branch is merged.

The only case in which the branch validation is, well, valid, is if no other change (of any kind) makes it into the trunk between when the validation started (assuming the latest trunk changes are merged into the branch at that time) and when the branch is merged back into the trunk.

If any other change makes it into the trunk while the validation is in progress it won't be taken into consideration, leaving room for breakage when the branch is merged into the trunk. To prevent that you'd have to merge again the trunk into the branch - to pick up the latest trunk changes - and re-run the validation.

Effectively this means the branch validations (or at least the successful ones allowing the branch to be merged into the trunk) need to be entirely serialized if you want to keep the trunk stable. Which may be manageable for a while if you keep a reasonably small number of such branches (maybe multiple devs working on the same branch?). How many branches? Depends on how stable you want the trunk to be: if you want to always pass your integration tests, your serialized validations would need to run the integration tests. At 40 min each you can support max ~60 branches (assuming the merges are automated and can be performed 24x7, which isn't trivial). Without automated merges the number of branches will be a lot smaller. This won't scale, the branch lifespans will gradually increase, diverging more and more from CI and will eventually wear down your team.

The same reasoning explains why devs shouldn't actually have to run all tests if they are too lengthy - by the time they complete them the results may be invalidated by other changes committed to the trunk. Also - results tests done by developers are often not that relevant ("But it built fine in my workspace!").

There are only 2 possible approaches (that I know of) to get relevant validations:

• the ones performed by the traditional CI tools - performed after the changes are committed to the trunk, reflecting the actual trunk reality, with all changes integrated. The only problem is if these validations fail - effectively blocking further development on the trunk until the culprits are found and the regressions fixed. Which may be (very) difficult in larger scale projects - as you already noticed.

• the ones performed by gating CI tools - only 2 out there that I know of: Zuul and ApartCI (which is my baby). These validations are performed prior to merging changes into the trunk, but in an orchestrated manner, ensuring they remain relevant - taking into consideration all other changes in flight that will eventually be present in the trunk by the time a particular change is merged into the trunk. The actual change merging into the trunk is entirely automated, executed by the CI tool itself, immediately after the relevant validations are completed successfully. These are, IMHO, the only ones capable of guaranteeing trunk stability in larger scale projects.

Pooling resources is a good thing - give them to the CI tool - that's how you maximize their usage.