I'm a Dev in a big (40+ Devs) old (16+Years) SW project. We still work with SVN as source control but we are getting after automation as much as we can.
One problem that hounts us is, that 40 devs check in stuff into the trunk without a fair chance of running all the tests. Our test suites consists of JUnit unit tests, Junit integration tests that run inside our app server and test business processes and some test tool blackbox tests that run after the deployment. Unittests run 10Min, Integration tests 40Mins and test tool tests run all night long.
You can imagine that for each check in, not ALL the tests and most of the time also not all of the Integration tests are run. Resulting in regular test failures on our CI environment (thats the good part: we have one!) But with 40 devs there is to much going on at the same time, so nobody knows for sure why a test failed. "Was it me? I changed something at another part of the app. This could not be the case, right...?"
I try to promote a branch based dev process with short(!) lived branches that could be testet before merging them into the trunk. Switching to Git is part of it but I would not be religious about it. Goal is that the trunk is always green. But for this I need an infrastructure that can support up to 60 branches.
I'm dreaming up a pooled build server system, where each team could request that a certain branch (maybe in a given revision/checkIn-ID) should be build and tested (maybe with a given testsuite). All the build requests get into a queue and are processed by the build server fleet. (Currently we have the problem that not enough hardware is available to scale the build infrastructure. Pooling the available hardware seems to be the best approach in my mind.)
What do you think about this approach? How would you do this? Is https://jenkins.io/doc/book/pipeline/multibranch/ the right path down the road?
Am I going down the wrong path here? I know that CI says that everybody checks in into trunk at least every day. But, damn, there are so many side effects when 40 devs check in together and the software (especially the data model) does not prevent side effects between parts of the application. We have to start somewhere. But where?