7

In a CI context one of the commonly-used measures of increasing the quality levels of the integration branch is a mandatory set of pre-commit quality verifications (typically including building some artifacts, performing unit tests and even some feature/integration tests).

Yet some regressions (build breakages, various test failures) are detected by the CI system verifications in exactly the areas which were supposed to be covered by these mandatory pre-commit verifications.

During analysis of these regressions an argument often heard is that the developer who committed the change identified as root-cause of the regression has successfully passed all such verifications. And often the claim is supported by hard evidence indicating that:

  • after the final version of the change was reached it was ported to a fresh workspace based on the tip of the branch
  • the required artifacts were built from scratch (so the build was totally fine, no cache-related issues, etc.)
  • all mandatory tests passed, including those covering the area in question and should have detected the regression
  • no intermittent false-positives affected the respective verifications
  • no file merges were detected when committing the change to the branch
  • none of the files being modified was touched by any other change committed in the branch since the fresh workspace was pulled

Is it really possible for a software change to cause such a regression despite correctly following all the prescribed processes and practices? How?

  • By pre-commit, you mean before entering the CI system (on the dev workstation) or SCM hooks checks ? I see a case on dev workstaion, but not on pre-commit hooks. – Tensibai Mar 7 '17 at 8:55
  • I guess it depends on the exact SCM system. Yes if the pre-commit hook executions for 2 changes going into the same branch can overlap in time - as it's the case for git, for example. – Dan Cornilescu Mar 7 '17 at 13:55
  • Sorry I was unclear, I mean does your pre-commit checks run on dev workstation or on the SCM side (central repository) ? All in all, I wonder if you're talking about checks a dev will launch himself before commit or if it is an automated testing system on 'receive' by the central server. – Tensibai Mar 7 '17 at 13:57
  • Yes, that's where I'm heading. With the note that the regression can happen even with centralized checks - if they are allowed to overlap in time. – Dan Cornilescu Mar 7 '17 at 14:03
  • Ok, rhetorical question then (I mean, one where you already have the answer and know what you're waiting). That doesn't leave much place for real answers at all. – Tensibai Mar 7 '17 at 14:09
5

There's one possibility I can think of, if when the dev are working on their own workstation, with sometimes images baked for virtual box to run on their workstation where your infrastructure doesn't use the exact same image.

The dev will need, while developing a feature, need to add a JVM parameter or whatever change to the middleware early in its work and forget it.

Before commiting, all unit/integration tests run on its workstation works great, as the baked image is shared, it works on every develloper system.

But when going through CI, it fails because the change to the middle-ware wasn't implemented, either because the dev forgot to ask for it, or just because the team in charge of updating the base images/provisioning system didn't had the time or did forget to update the system.

That's a good thing it break in CI, because it tells early before going into production that the system won't work as expected, but sometimes it becomes a hell to find the missing parameter.

This last point advocate to avoid rejecting commits, and just break on CI on a feature branch, thus it won't block anyone else, and let the dev fix the problem early, when the change is needed and prevent this change to be forget in the flow.

FWIW, we did exactly this here, developers had whole access to development machines and releases in Q/A were failing because a parameter change has been forget, we did move to chef to handle the configuration of the middleware (tomcat now) so each needed change to the infrastructure has to be coded somewhere and will be reproduced in all environment.

  • Bonus points if the changes made by developer rely on a function from the testing framework so it works everywhere but the production environment (which I managed to do once). – Jakub Kania Nov 24 '17 at 23:24
2

Sure it is. Production is always different. Real money. Real load. Real users. Real pain. This is why it is so important to put any significant change behind a feature flag. Your deployment should not change anything. Turning on a feature is the only thing that should make significant changes to your site.

  • I'm talking about regressions caught by the CI system - in the integration branch. Before deployment in production. – Dan Cornilescu Mar 7 '17 at 4:57
  • Consider a CSS change that passes all tests but in production it is noticed that sales drop to $0. Upon visual inspection it is noticed the button text is now the same color as the background and users refuse to click on a colored blob. Automation can pass this type of problem unless you have UI regression and people actually verify the regressions. CI may not be where this problem appears. – No Refunds No Returns Mar 7 '17 at 5:01
  • Also covered: "all mandatory tests passed, including those covering the area in question and should have detected the regression" - the same test passed in pre-commit verification of the developer's image but failed with the CI image built after the change was committed. – Dan Cornilescu Mar 7 '17 at 5:04
2

The breakage is always theoretically possible because the pre-commit verification performed by the developer is done in isolation and thus can't take into account other in-flight changes being verified in parallel. Such changes can interfere with each-other and cause regressions without actually have collissions detectable at the SCM-level.

A simple example of such interfering changes:

Let's assume the code in the latest version of a project branch includes a certain function, defined in one file and invoked in a couple of other files. Two developers working in parallel on that project are preparing to make some changes to the code.

Developer A reworks that function removing or adding a mandatory function argument and, of course, updates all invocations of the function in all the associated files to match the updated definition.

Developer B decides to add an invocation of said function in a file which didn't contain any such invocation before and is thus not touched by developer A's changes. Of course developer B is filling the argument list to match the function's definition visible in the latest label. Which is the old definition, as developer A's changes aren't yet committed.

Both developers correctly perform the pre-commit verifications with a pass result and proceed to commit their code changes. Since the two changesets do not touch the same files no final merge happens, which typically would be an indication of potential problems, warranting a closer look and maybe a re-execution of the pre-commit verification. Nothing whatsoever can give even a subtle hint that something may go wrong.

Yet the end result is catastrophic - the build is broken as the function call added by developer B doesn't match the function definition updated by developer A.

1

When you find this type of issue, you should write some new extremely quick running acceptance tests that can eke out these issues, and add them to your build verification tests that run prior to your integration tests. You should constantly be shifting left and trying to shorten the feedback loop to the developers committing changes. If you cannot find a way to do this, perhaps your architecture isn't as agile as it needs to be.

@Dan Cornilescu - your scenario is valid for tightly coupled architectures which is why loosely coupled architectures (microservices of versioned RESTful APIs) have emerged as the current best practice in high performing organizations. These matrix of services organizations have other complexities to overcome though.

Sometimes you need to refactor your entire architecture to overcome issues such as these. I believe it was both Google and eBay that have completely rearchitected their platforms 5 times (in a span of something like 10 years) due to constraints their previous architecture imposed upon them.

  • I think you missed the point of the question: the mandatory set of pre-commit quality verifications mentioned in the question already include the acceptance tests you're talking about. They're executed by devs prior to their commits and pass and they're executed by the CI system and they fail after both commits are in. – Dan Cornilescu Mar 24 '17 at 1:15
  • That was my point, perhaps a completely new architecture is the appropriate response to this issue that cannot be resolved by the current architecture. – icewav Mar 24 '17 at 1:29
  • you mean product architecture or process architecture? – Dan Cornilescu Mar 24 '17 at 1:30
  • A new product architecture if the current process architecture can't solve the issue. – icewav Mar 24 '17 at 1:51
  • The product architecture is irrelevant, the question applies to pretty much any product architecture, except maybe for very small, trivial sw products. See the example in my answer. – Dan Cornilescu Mar 24 '17 at 1:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.