10

Traditionally CI systems only perform monitoring of the quality levels in an integration branch, by performing QA verifications on the codebase where the changes are already committed, watching for regressions and sending notifications for human intervention.

But when these regressions are detected the branch has already been in trouble at least since the respective QA verification started and will remain in such state (or even get worse!) until all the culprits are identified, repairs for them committed and a new QA verification confirms the branch quality level was restored. The branch can be considered blocked for normal development during all this time.

Is there a CI tool capable of actually preventing such regressions from happening, which would perform pre-commit QA verifications and allowing commits only when the codebase updated with the respective commits would be passing those pre-commit QA verifications as well, thus guaranteeing a minimum branch quality level?

Update: the assumption is that suitable automated QA verifications with appropriate coverage to be able to detect the respective regressions are available for invocation by the CI tool(s).

Improve this question
2
  • I keep wondering about the correct way to understand this question (and your own recommendation in your own answer). If I'd replace "monitoring" with "after the facts", and "preventing" with "prvent them from happening", would that more or less be the same question then? Also, maybe you can add some example scenario to explain the difference?
    – Pierre.Vriens ♦
    Commented Mar 29, 2017 at 8:31
  • @Pierre.Vriens Is this any better?
    – Dan Cornilescu
    Commented Mar 29, 2017 at 12:27

5 Answers 5

Reset to default
6

As far as I can tell, you're looking for a tool that will reject commits that break the build—a CI tool probably won't be able to prevent regressions by actually fixing your code, but it can stop you from adding bad code to the repository.

Atlassian has a few interesting applications of Git hooks:

Server-side pre-receive hooks are an especially powerful compliment to continuous integration because they can prevent developers from pushing code to master, unless the code meets certain conditions – like elite ninja guardians, protecting it from bad code.

If you're using Git, the hooks are very powerful (and there are similar hooks for SVN, Mercurial and other version control systems), and you might find it useful to use them to run pre-commit checks.

The Git documentation has a page on creating a hook to reject pushes if they don't meet a certain criteria which could easily be adapted to this use-case.

However, a lot of people would argue that rejecting commits is a bad idea on a feature branch—you'll just be wasting time fighting against your CI system when the build breaks for some reason, instead of actually fixing the bug.

On the master branch, it could make sense to reject broken merges, because you might want to ensure it always builds. For a feature branch, you will inevitably break things, and since the code isn't going into production now, it makes more sense just to warn you than actually reject your commit altogether.

Improve this answer
4
  • Well, what good is a sw image which builds, but is DOA from testing prospective? Devs can't test their code changes even if they build, so they'd be just as blocked. So in general I'd extend commit rejection to anything that fails a minimum QA check, chosen by balancing 2 conflicting requirements: as high as possible to maximize the number of devs protected from blocking and as low as possible such that the QA verification delays don't slow the process down too much.
    – Dan Cornilescu
    Commented Mar 3, 2017 at 13:39
  • An example of this might be the pull request model where you can pose certain restrictions on whether a pull request can be merged or not. For example, we (my company) uses Atlassian Bitbucket Server and there are options to require at least N number of approvals and X number of passing builds for the given branch before a pull request is allowed to be merged. This doesn't outright reject it. But prevents accidental merging when tests fail or other eyes have not seen the code yet.
    – Andy Shinn
    Commented Mar 4, 2017 at 4:41
  • @AndyShinn: Yeah, I find that quite useful—GitHub also offers protected branches and checks on pull requests, which are often helpful.
    – Aurora0001
    Commented Mar 4, 2017 at 11:19
  • 1
    One argument for allowing broken code in feature branches is it allows devs to push code they are working on to the repo, even if it's not quite ready yet. This can be useful for sharing code with others for early code/architecture reviews before things are ready to go, help debugging issues, or for someone who is leaving on vacation to put partially done work where others can get to it. For feature branches, I would only put things like linters and as pre-commit hooks.
    – bradym
    Commented Mar 4, 2017 at 19:18
2

No tool could possibly guarantee no regressions - that depends much more on your tests than the tool executing them. However, you can help to prevent regressions that will be caught from entering into the integration branch. You can do this with pre-commit hooks, but it's often easier with pull requests (which hopefully you're already using for peer code review).

If a branch is up-to-date with its upstream (where the PR is merging to), and its tests pass, then they will still pass after the merge; the state of the target branch after the merge will match the state of the source branch before the merge.

It's generally not particularly difficult (depending on the tools used) to indicate both whether the source branch in a PR is up-to-date with the target, and if it has a passing CI build. You can use this as a requirement (by policy, and/or enforced in software) for merging the pull request.

Improve this answer
6
  • "If a branch is up-to-date with its upstream (where the PR is merging to), and its tests pass, then they will still pass after the merge" - Why? A merge is a discontinuity, it brings unknowns. Like conflicts - the code might not even build until they're resolved. You need to run the tests and confirm that they pass for any branch merge. I'd say even for a fast-forward one, if you want to play it safe. See apartsw.com/insights/2016/11/16/…
    – Dan Cornilescu
    Commented Mar 4, 2017 at 3:57
  • Um, yes, such guarantee is possible, check my answer. Well, maybe I should clarify: by regression I mean results worse that the branch baseline results (and ignoring possibility of false positives, those need to be addressed as they can skew the baseline, derailing the whole thing and requiring human intervention). Intermittent false negatives are just a nuisance, slowing things down, but can be dealt with.
    – Dan Cornilescu
    Commented Mar 4, 2017 at 4:16
  • You can't guarantee no regressions, you can only guarantee no detected regressions. If a change causes a regression not caught by a test, that's a regression that a CI tool cannot make any guarantees about. And while merging two sets of changes does bring unknowns, you can choose to do that in the feature branch (by merging the upstream in) before merging the other direction. If the source is up to date with the target, it's a simple merge (fast forward), and afterward the target state will be identical to the source state before the merge, hence if tests passed before, they'll pass after.
    – Adrian
    Commented Mar 4, 2017 at 4:30
  • Splitting hairs. The CI tool can be configured with a test to detect and thus protect against whichever regression you care about. I won't argue too much about the merges - my goal is to avoid them as much as possible, they're just trouble overall :)
    – Dan Cornilescu
    Commented Mar 4, 2017 at 4:46
  • My point is that it's not the CI tool offering that protection, it's you, by writing the tests. The CI tool can't provide any guarantees beyond the tests you provide.
    – Adrian
    Commented Mar 4, 2017 at 16:49
1

True continuous integration tools (as opposed to just continuous testing) like Reitveld and Zuul can help, though they are only as good as the tests you write and code reviews you do.

Improve this answer
4
  • But Reitveld appears to be a collaborative code review tool, not a CI tool, am I missing something? This is what I looked at: github.com/rietveld-codereview/rietveld
    – Dan Cornilescu
    Commented Mar 3, 2017 at 7:09
  • Zuul appears to be indeed, related, I'll study it closer.
    – Dan Cornilescu
    Commented Mar 3, 2017 at 7:16
  • It doesn't do testing but it does handle some aspects of branch management, acting as a gatekeeper system, which is the best bet to keep bad code from getting in by blocking the merge.
    – coderanger
    Commented Mar 3, 2017 at 7:37
  • I see what you mean. Well, it can help with the overall code quality, but by itself doesn't bring any guarantee. Two perfectly good changes that pass all QA verifications independently can still cause breakages when they meet in the branch.
    – Dan Cornilescu
    Commented Mar 3, 2017 at 14:13
1

Use GitLAB, you can set in project-settings to only allow a merge when the pipeline succeed, so can have a truly Continuous Integration, combine that with adding your QA to the list of merge approvals and with Dynamic Environments, you can have quality assurance before you merge to the master.

Improve this answer
1
  • That works but only if you do not allow a 2nd merge to start QA checks before the previous merge is completed, otherwise the 2nd merge won't see the 1st one, leaving room for regressions. In other words the merges (including their QA checks) must be completely serialized, which doesn't scale for large teams.
    – Dan Cornilescu
    Commented Sep 17, 2018 at 9:28
0

ApartCI is a CI system designed exactly to prevent regressions, thus guaranteeing flat or increasing branch quality level. Still in beta.

It orchestrates centralized pre-commit verifications in such manner as to ensure that a change is committed only after it is verified, together with all other changes committed before it, to meet or exceed the latest branch quality level.

This is the key difference compared to traditional developer-driven pre-commit verifications, often done in parallel , which leaves room for regressions caused by interfering changes which were never tested together.

The tool is also designed to easily scale - capable of sustaining very high rates of incoming candidate changes and supporting 100s/1000s of developers working in the same integration branch.

Disclaimer: I'm the author of the tool and founder of the company offering it. Apologies for the ad.

Improve this answer
6
  • It's good that you added the disclaimer, but personally I consider asking a question and self-answering it by promoting your own company or products to be a form of spam.
    – THelper
    Commented Mar 4, 2017 at 11:39
  • I've asked a question on meta whether this is allowed or not: meta.devops.stackexchange.com/q/59
    – THelper
    Commented Mar 4, 2017 at 12:04
  • SnapCI did this too. RIP.
    – paul_h
    Commented Apr 1, 2017 at 23:21
  • @paul_h, unless I'm missing something SnapCI and its recommended replacement GoCD are both based on post-commit verifications (triggered by polling the SCM), so they're still monitoring-only. Except maybe for PR checks, but unless these checks are completely serialized they can only reduce the regression occurence rates, but not eliminate them completely.
    – Dan Cornilescu
    Commented Apr 2, 2017 at 3:31
  • Dan, not polling no, hooks. And to a short-lived PR branch that is not yet merged into trunk/master - trunkbaseddevelopment.com/game-changers/…
    – paul_h
    Commented Apr 8, 2017 at 19:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.