So I have a Jenkinsfile defining a build pipeline and then a Jenkins job (not pipeline) with a very simple deployment script for our Docker stacks.

Seeing that Jenkinsfiles can become as complex and powerful as one's coding skills are (seeing that they can groovy fairly well), would I have everything in one Jenkinsfile or have multiple Jenkinsfiles?

Consider this theoretical process:

> Build artefact > Build Docker image
> Deploy Docker container to testing > tests & QA
> deploy to staging > have client check it
> deploy to production

Would all of that go into one Jenkinsfile and run different steps of the pipeline according to user input like

"which stages should be performed?
Please specify:
Build artefact,
build Docker Image, ..."

This would result in a complex Jenkinsfile but you'd only have one single file.

Or should this process be split into multiple Jenkinsfiles with multiple Jenkins jobs/pipelines for each Jenkinsfile to do a part of the process?


It really comes down to personal preference.

One additional tool you might not be aware of are shared libraries for Pipeline. These allow you to quickly write custom Pipeline steps or factor out common Pipeline code without writing a Jenkins plugin in Java.

Between multiple Pipeline jobs and shared libraries, there are many ways to split up your job's code and there's no one right answer on how to do it. One suggestion I have is to determine what steps in your process are "atomic" - that is, what are the smallest group of steps that you envision admins/ops/devs/etc. might need to run on their own? Each "atomic unit" should then become its own job.

For instance, let's say you're automating a deploy process, and your process in a general sense looks like this:

  • build
  • deploy to dev
  • deploy to prod

Then I would create three jobs, one for each bullet point. This gives you some nice advantages:

  • You can re-run each job individually in case something fails instead of having to restart the entire process over from the very beginning. Deploy to prod failed? You only need to re-run the deploy to prod job. You don't need to rebuild or re-deploy to dev.

  • Each stage in the process is separated out so you can see how often it succeeds or fail. Some stages in your process may be more robust or fragile than others; this kind of separation gives you insight into that.

  • This level of abstraction makes it easy for non-ops people to perform ops tasks (if that is something you want/need at your org). When everything is monolithic, the only thing non-ops people can do is run the entire process from the beginning. When everything is split up into tiny jobs, you need intimate ops knowledge to know which pieces to run in what order. By separating your jobs into independent, easy-to-understand, appropriately bite-size chunks, a non-ops person only needs to press a single button to kick off an automated ops process.

  • So I reckon a definite pro argument for mulitple jobs (instead of one large job) would be that you can set permissions for who is allowed to execute a job, but you can't set permissions on who is allowed to execute certain steps of a build pipeline. So in addition to your argument, which I find very valid, the Jenkins permission scheme also speaks fo rmultiple jobs. And then it would be: One Jenkinsfile for each job that is defined in a Jenkinsfile. (Seeing as there can be jobs that are not configured with a Jenkinsfile but via the Jenkins GUI). – Worp Mar 13 at 6:11
  • This is a good answer, I will leave the question open to see if other arguments are made that I am overlooking, but this seems fairly strong a point. – Worp Mar 13 at 6:14

Having been down a similar road, here are my suggestions:

Break your tasks apart by function; build is different to deploy, which is different to release. You can get more granular, but let's stick to the basics for now...

Secondly, if you want to consider who is permitted to do certain actions, you might break these functions apart by team (dev, test, release).

Build pipeline:

  • This is probably your dev team.
  • A single job to check out sources, compile and package.
  • Perhaps run unit tests against this package
  • Perhaps send the code to SonarQube for analysis
  • If successful, publish this build artifact somewhere, like Artifactory/Nexus
  • I automatically run this job whenever someone pushes to the repo
  • Additionally, I have some rules which limits behaviour based on the branch (I don't really want to publish builds for every single feature branch, but always for release branches)

Deploy pipeline:

  • This job should be be generic, and accept parameters that gives it context
  • Are we deploying to a dev server? Or to Test/Pre-rel/Live?
  • Create a deploy-dev job, and a deploy-prod job. Both use the same Jenkinsfile
  • Who has permission to perform this action?
  • What are we deploying? This is a dropdown of artifacts available in Artifactory.
  • I use JobDSL to generate multiple jobs that use the same pipeline. One for dev/test/pre/live. You could manually create these jobs at first with appropriate deploy targets/users

The above lets me write highly targeted and specific pipelines, yet gives the teams the flexibility they need for day to day stuff. Dev and Test are self-service. Only the release team has access to deploy to pre-rel/live.

Other pipelines:

  • Once that's working, you'll probably look at jobs that can bounce servers (self-service for teams) and promote builds from snapshot/staging/RC repos to gold repos in Nexus/Artifactory, etc.
  • Are you using git-flow? How about jobs to create/finish release/feature branches, bump maven versions, tidy up stale branches, perform integration builds, etc. All good fun.

Hope this helps.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.