I recently built a build-test-publish-deploy-rollback pipeline and struggled with the question of one-or-many pipes. I decided on a single pipe to do all tasks, with the reason that it would be easy to go to one place, and tell the pipe what you want to do by entering parameters. It seemed like a good idea at the time, but I have smelled a few drawbacks since releasing it:
- A success or failure of a given pipeline build doesn't mean the application code is necessarily bad. For example, we had a push to production fail due to a bug in the pipe code, but that doesn't mean the app build is bad. But the Build Status Badge shows
Failed in our ReadMe. Which isn't true (the way we see it).
- It is more complicated. There are several paths through the pipe, determined by the parameter values. More chance for error, harder to review, etc.
- Blatantly breaks the principle of single responsibility, as well as the principle of least amazement. It would not necessarily do what one might expect based on some rules i'd built in, for the purposes of protecting the users. With a simpler pipe, there'd be less chance of error.
- Our org isn't evolved to true Continuous Deploy yet, so there will always be a delay and a person pushing the "deploy" button, even to QA. We may move to CD into QA sooner rather than later, but we have to prove the safety and reliability of all of this before the higher-ups will be comfortable with CD into production.
All of which speaks to separate pipes with clearly delineated functions.
I am now building another pipeline for a service of the same type, and I plan to refactor to use separate pipes to build and deploy - same as @jayhendren listed above:
- Build, Test (barely exists at this point), Publish Artifacts
- I am trying to build cookbooks to spin up temporary machines to deploy to give devs siloed test sandboxes.
- Deploy to QA
- Deploy to Prod
One thing worth mentioning is we are definitely using Multibranch pipes, at least for the build pipes, and Declarative syntax wherever possible.
Thinking about it as I write this, the Build pipe can stay as it is now, as a multibranch pipeline job built off of the Jenkinsfile living in the app's repo, and building on any push to github.
Then I can manually build additional pipelines for deploying - templated so they're generic, and only pass the unique variables to each one.
I want to come back to this and contradict myself. After more internal discussion on the design, and my own growth as a Jenkins user, I went back to a single pipe for all tasks.
The difference was using fewer parameters, and controlling the flow with the branch/PR that triggered the build. This way, although the tasks may vary, a pipe's output is based on its properties only.
Obviously this reduces the overall complexity by only having one pipeline, although the single pipeline's conditional checks may become more complex to handle all the conditions.
I should also add that doing the non-code design work like flow charts and diagrams helped expose the possibilities - going straight to code hid the big picture from me at first.