Is there a way to use a Jenkinsfile from a git-submodule in a multibranch pipeline?

Question

Background: The nature of our project's source involves multiple long-living branches, each one has its own Jenkins job (manually created).

Multibranch limitation: Unfortunately we do not use a Multibranch Pipeline to organize the project, because when (for example) a change is needed in the Pipeline, it would need to be added (manually) to each of the long-living branches' Jenkinsfiles.

Current implementation: We store the Jenkinsfile in a separate repository (let's call it ci-repo), which is added as a submodule to the project's repository (let's call that dev-repo).

What still hurts: We need to manually monitor branches of the dev-repo and manage job creation. Multibranch pipelines support submodule checkout for source files, but branch discovery requires presence of a Jenkinsfile under the parent repository (in our case dev-repo).

The question: Jenkinsfile from git-submodule of Multibranch Pipeline - Is there a way to do this? Perhaps using an initial/mock Jenkinsfile in the dev-repo, with one stage in which is calls the Jenkinsfile from the submodule? Can a Jenkinsfile be over-written or replaced somehow during runtime?

Answer 1

I agree with Omri that you are heading the wrong way. I would recommend using a shared library which implements the pipeline. You would then have a simple Jenkins file which looks all the same. See https://jenkins.io/blog/2017/10/02/pipeline-templates-with-shared-libraries/

Let's say you have a shared library which contains a vars/commonpipeline.groovy which contains something like this:

def call(body) {
    // evaluate the body block, and collect configuration into the object
    pipelineParams= [:]
    body.resolveStrategy = Closure.DELEGATE_FIRST
    body.delegate = pipelineParams
    body()

    pipeline {
        agent {
            label pipelineParams.nodes
        }
        options {
            buildDiscarder(logRotator(artifactNumToKeepStr: '1', numToKeepStr: '10'))
            disableConcurrentBuilds()
            timestamps()
            timeout(time: pipelineParams.timeout, unit: 'MINUTES')
        }

        stages {
            stage('Build') {
                steps {
                    //TO SOMETHING
                }
            }
        }
    }
}

Then you can have a Jenkinsfile in your dev project which loads the library and uses the common pipeline like this:

import org.apache.commons.io.FileUtils
library identifier: 'shared-pipeline-library@master', retriever: modernSCM(
  [$class: 'GitSCMSource',
       remote: 'https://bitbucket/jenkins/hared-pipeline-library.git',
       credentialsId: 'bitbucket.service.user'
  ])

commonpipeline {
    nodes     = 'BUILD' /* label of jenkins nodes*/
    timeout   = '60'
}

Answer 2

There may not be a way to make your idea work with submodules. When a Jenkins multibranch pipeline polls SCM for changes, it has no visibility into submodules. I verified this recently. The result in "Scan Multibranch Pipeline Log" was a Communication error for url response. The Jenkinsfile was not found, and so no build was triggered.

The good news is that I discovered a straightforward way to achieve what we're after without using submodules. Have a look at the Remote Jenkinsfile Provider Plugin.

When a pipeline is polling a repository to trigger new builds, it looks for two things for each branch:

1. The polled branch has new SCM changes since the last build (HEAD is different).
2. A Jenkinsfile exists in repository X on branch Y at path Z.

In a multibranch pipeline's Build Configuration section, normally only one Mode is available to tell Jenkins where to find the Jenkinsfile: "by Jenkinsfile". This mode allows us to specify Z (path) only; X and Y are implicitly set to the repository and branch being polled.

What the Remote Jenkinsfile Provider Plugin gives us is a new Build Configuration Mode: "by Remote Jenkins File Plugin". This mode allows us to explicitly set the repository, branch, and path for the Jenkinsfile - X, Y, and Z. In your example, X would be ci-repo, but any Jenkinsfile in any repository is fair game. You could have ci-repo be a submodule of dev-repo if it's convenient, but no such relationship is required to make this work.

If desired, a "Match branches" option uses the polled branch name for Y, looking for a remote Jenkinsfile on a branch of the same name. For example, while polling dev-repo/feature-123, it would look for a Jenkinsfile on ci-repo/feature-123.

As long as the Jenkinsfile exists at the specified location, the second criteria for triggering new builds will be satisfied. You'll see something like this in the pipeline's polling logs:

Looking up DR/dev-repo for branches
Checking branch master from DR/dev-repo
No local file defined. Skipping Source Code SCM probe, since Jenkinsfile will be provided by Remote Jenkins File Plugin
    Met criteria
Scheduled build for branch: master

Instructions for how to set this up, with screenshots, can be found on the plugin's page.

Caveat: Pushing changes to a remote Jenkinsfile (ci-repo) will not trigger any builds of the polling repository (dev-repo). If this behavior is desired, one solution would be to use webhooks. Another would be to create a pipeline that polls the remote Jenkinsfile's repository and builds other pipeline jobs as desired using the build step, like so:

build(job: "dev-repo/${BRANCH_NAME}", wait: false)

Answer 3

I believe you are heading in the wrong direction. The way to implement what you want is to separate the "devops infrastructure logic" (placed in the 'ci-repo') from the pipeline trigger - a jenkinsfile inside the dev-repo, which includes the "devops infrastructure logic" as a tagged artifact.

Pros:

• You have a central place to develop (and test) devops code
• When changes are backward compatible, you can update all pipelines by moving the tag
• When changes are NOT backward compatible, repos & specific branches can gradually move to a new tag
• There is room for specific repos & branches to create their unique process if they have to

Cons:

• Yes, sometimes people will need to update Jenkinsfiles in multiple places as they are accepting a new version. I actually think it's good.

Answer 4

We have the exact same problem, and wanted to run a multibranch pipeline using a Jenkinsfile located in a git submodule. Our setup is a bit different because the submodule contains also libraries that are shared amongst several projects. So this is more than a ci-repo, but the principle stay the same. We prefer this approach because it allows linking easily the version of the project to the version of the pipeline that was used at release (instead of manually pointing to some revision of a shared library).

Our solution is actually a direct application of https://stackoverflow.com/questions/37800195/how-do-you-load-a-groovy-file-and-execute-it.

Let's say we have following project tree:

build/
    Jenkinsfile
src/
    ...
common/           <-- a git submodule
    src/
        ...

Our Jenkinsfile is something like

    // file: build/Jenkinsfile

    def doThings()
    {
        // ...
    }

    try {
        node {
            // ...
        }
        // ...
    }
    catch(e) {
        // ...
    }
    // ...

So basically some utility functions and some pipeline stages (here in a try-catch for instance).

Two actions are required to move this file to our common submodule:

• Move the file to submodule and adapt it slightly.
• Replace the original jenkinsfile with a small boilerplate version.

Our moved Jenkinsfile becomes

    // file: common/build/Jenkinsfile

    def doThings()
    {
        // ...
    }

    def call() {
        try {
            // ...
        }
        // ...
    }
    return this

So basically we surround the pipeline in a call function, and don't forget to return this at the end of the file.

The original Jenkinsfile is then replaced with this small boilerplate code as explained in the original post:

// file: build/Jenkinsfile

def myPipeline
node {
    checkout scm // mandatory
    myPipeline = load "common/build/Jenkinsfile"
}
myPipeline()

Of course this can be tuned further to pass parameters, use several jenkinsfile, etc. For instance reusing ideas from the documentation:

def call(Map params)
{
    println "Building for project " + params.projectName
    // ...
}

// ...
myPipeline(projectName: "MyProject")

Also note that this checkout the whole project, which might be inefficient. A better approach would be to checkout only the submodule that contains the Jenkinsfile.