2

Ideally, a team should be able to run all the pipelines on every commit and automatically deploy to production.

However, I don't understand how to most effeciently use our resources so that hours are wasted running tests on code which is about to be overwritten by the next commit.

If a build or test takes 20 minutes to go through the pipeline, the system gets "clogged".

What mechanism is used to solve this problem? Do tests/jobs get canceled as new code gets pushed? If so, then how do you deploy multiple times a day? Add to this the need for Pull requests or sign off. What intermediate steps can be taken until it's possible to automate everything?

2 Answers 2

5

We break ours down into stages, and each stage pulls, rather than pushes.

Our build takes 20 minutes. So if I commit once a minute, I will still only get three builds an hour, but each build will contain 20 commits.

The tests then pull on the latest successful builds and run them, and the same for the deployment scripts.

The main point is that we do NOT build every commit individually. We do NOT test every build individually.

I probably got all this from Continuous Delivery by Jez Humble and David Farley.

1

The typical CI/CD pipeline is normally triggered for a branch by new versions being created on that branch, which are the results of branch merges, either manual or automated by some tools, like PRs (or of direct commits), the traditional CI tools don't need to do any code merging.

Once triggered, the pipeline execution attempts to complete as many steps as possible, provided, of course, that the necessary resources (build servers, testbeds, etc.) are available. If the pipeline is triggered again because of new code changes coming in before the previous execution(s) complete you'd just have new pipeline runs, in parallel with/overlapping the ones in progress. Not a problem. Whichever run is successful will perform a deployment, regardless of how much time passed since the previous one.

The runs already in progress are not waste, they still provide crucial information about the quality of the changes that got integrated into the branch, which is, after all, the primary goal of CI/CD. Stopping a particular pipeline run N because N+1 is triggered would prevent you from immediately identifying the culprit if the N+1 run fails - you can't tell if it's because of the N or the N+1 commit. You'd also unnecessarily delay deployment of the N commit (if its run would have been successful) - you now need to either rollback or fix the problem and do at least one more full pipeline run to deploy it.

The clogging you mentioned only happens if you don't have enough resources to meet the demand. Mostly a planning matter. If your resource deficit is very small your pipeline may still function close to ideal, with just small delays during peaks of activity but catching up during idle/low activity periods.

If the deficit is large and the pipeline indeed doesn't keep up with the rate of the incoming changes you'll be forced to diverge from the CI way, with multiple changes being picked up in a run, thus losing the ability to trivially identify the changes causing regressions. The longer it takes to identify the culprits, the longer your branch stays blocked, the slower your overall average pipeline speed gets. Fundamentally this is no different than running your pipeline only nightly, for example, the only difference is just the number of changes picked up in every run.

As for "automate everything", I'm afraid that's just a dream in traditional CI - even if done well it can only indicate when regressions in the branch quality occurred, but human intervention is still required for repairing and unblocking the branch. AFAIK only the gating commit CI systems can be entirely automated, but those are different beasts.

3
  • What is gating CI? And what happens when N+1 commit finishes before the N commit?
    – avi
    Commented Dec 28, 2019 at 17:30
  • If N+1 is successful it can be deployed. Even at that point the N run isn't a complete waste, if it fails it would indicate a refpoint that should not be used. But if it is successful it should not be deployed anymore as it would rollback the N+1 change. Depending on the CI system this might be automatable. Commented Dec 29, 2019 at 9:02
  • 1
    The gating CI systems perform orchestrated centralized verifications before the changes are committed/merged into the branch. Only on successful verifications the changes are (automatically) merged into the branch, thus ensuring the branch remains always green and ready to deploy. Human intervention is no longer required during normal operation. Commented Dec 29, 2019 at 9:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.