In a CI context one of the commonly-used measures of increasing the quality levels of the integration branch is a mandatory set of pre-commit quality verifications (typically including building some artifacts, performing unit tests and even some feature/integration tests).
Yet some regressions (build breakages, various test failures) are detected by the CI system verifications in exactly the areas which were supposed to be covered by these mandatory pre-commit verifications.
During analysis of these regressions an argument often heard is that the developer who committed the change identified as root-cause of the regression has successfully passed all such verifications. And often the claim is supported by hard evidence indicating that:
- after the final version of the change was reached it was ported to a fresh workspace based on the tip of the branch
- the required artifacts were built from scratch (so the build was totally fine, no cache-related issues, etc.)
- all mandatory tests passed, including those covering the area in question and should have detected the regression
- no intermittent false-positives affected the respective verifications
- no file merges were detected when committing the change to the branch
- none of the files being modified was touched by any other change committed in the branch since the fresh workspace was pulled
Is it really possible for a software change to cause such a regression despite correctly following all the prescribed processes and practices? How?