CI/CD pipeline with manual advancement?

Question

I am currently learning about CI/CD pipelines for DevOps. I have some practice implementing them in some basic scenarios, but I've recently encountered a slightly more complex scenario and I'm not sure the best way to integrate CI/CD pipelines into this project.

The Scenario

The client and the development team have agreed to work in 2 week sprints. Once the sprint has ended, the client's management team wants 1 week to play with the sprint delivery in a sandbox environment (we'll call it UAT) before it is released to production (PROD). This means that releases to PROD happen on "off weeks" from a sprint perspective, if they happen at all for that sprint. If the customer is unhappy with something they'll ask us to not deploy to prod until the issue is resolved. We still continue working in sprints, we just don't release that sprint to PROD.

Question

Clearly the naive CI/CD pipeline approach doesn't work here since most code changes aren't expected to make it straight down a pipeline to PROD. In light of this, what is the best approach here for a team that is still interested in utilizing pipelines as much as possible?

Some Thoughts

• We could use a single pipeline with manual approval steps prior to the UAT portion and then again prior the PROD portion. This has the benefit of being a single pipeline, so a single build artifact progresses through the entire pipeline, which seems to be a best practice. The problem is that most things that enter the pipeline are going to get hung up at a manual approval step. Most pipeline runs will then either be stalled or marked as failed.

• We could use 2 pipelines. One is totally CI/CD with DEV and STAGING environments. Every two weeks, a separate pipeline would be manually started (or started by some other release mechanism associated with SCM like tagging or merging). This would be in charge of rebuilding, retesting, and deploying to UAT and PROD.

If it's relevant, jenkins would be the pipeline of choice.

Answer 1

The "D" in CI/CD means different things to different people, either delivery or deployment. For both groups, it rarely means release everything immediately to production. For the delivery group, the output of CI/CD is an artifact that can be deployed. For the deployment group, there are typically gates in pipelines with either automatic or manual checks before deployment happens. Automatic gates include the unit tests, CI tests, but also things like canary releases with monitoring of key metrics.

Manual approvals in Jenkins involve adding an input step to your pipeline. E.g.

input([message: "Ready to continue?"])

In my experience, having various stalled pipelines waiting for user approval is fairly common. I often include timeouts to clear out stale prompts, and sometimes I configure old prompts to automatically cancel with a more recent pipeline is approved to reduce the workload on users. If you do the latter, you may want to have a separate pipeline for backout/revert jobs since that would otherwise being a new job that gets promoted to production in an emergency and aborts all the other work in progress.

I still like to separate my builds from deploys with separate pipelines. It gives developers better feedback with their CI pipelines of what's working. Then the deployment team picks from a list of automatically queued up deployment jobs that were automatically triggered from those successful CI jobs.