2

Background: The nature of our project's source involves multiple long-living branches, each one has its own Jenkins job (manually created).

Multibranch limitation: Unfortunately we do not use a Multibranch Pipeline to organize the project, because when (for example) a change is needed in the Pipeline, it would need to be added (manually) to each of the long-living branches' Jenkinsfiles.

Current implementation: We store the Jenkinsfile in a separate repository (let's call it ci-repo), which is added as a submodule to the project's repository (let's call that dev-repo).

What still hurts: We need to manually monitor branches of the dev-repo and manage job creation. Multibranch pipelines support submodule checkout for source files, but branch discovery requires presence of a Jenkinsfile under the parent repository (in our case dev-repo).

The question: Jenkinsfile from git-submodule of Multibranch Pipeline - Is there a way to do this? Perhaps using an initial/mock Jenkinsfile in the dev-repo, with one stage in which is calls the Jenkinsfile from the submodule? Can a Jenkinsfile be over-written or replaced somehow during runtime?

  • I have a similar problem: multiple repositories have the same steps to execute build and test. We created a shared library with all the functions (such as test, build, deploy), but again the basic logic of the JenkinsFile must be stored multiple times in all the repositories. Did you a find a solution? – user1382278 Mar 20 '20 at 16:38
3

I agree with Omri that you are heading the wrong way. I would recommend using a shared library which implements the pipeline. You would then have a simple Jenkins file which looks all the same. See https://jenkins.io/blog/2017/10/02/pipeline-templates-with-shared-libraries/

Let's say you have a shared library which contains a vars/commonpipeline.groovy which contains something like this:

def call(body) {
    // evaluate the body block, and collect configuration into the object
    pipelineParams= [:]
    body.resolveStrategy = Closure.DELEGATE_FIRST
    body.delegate = pipelineParams
    body()

    pipeline {
        agent {
            label pipelineParams.nodes
        }
        options {
            buildDiscarder(logRotator(artifactNumToKeepStr: '1', numToKeepStr: '10'))
            disableConcurrentBuilds()
            timestamps()
            timeout(time: pipelineParams.timeout, unit: 'MINUTES')
        }

        stages {
            stage('Build') {
                steps {
                    //TO SOMETHING
                }
            }
        }
    }
}

Then you can have a Jenkinsfile in your dev project which loads the library and uses the common pipeline like this:

import org.apache.commons.io.FileUtils
library identifier: 'shared-pipeline-library@master', retriever: modernSCM(
  [$class: 'GitSCMSource',
       remote: 'https://bitbucket/jenkins/hared-pipeline-library.git',
       credentialsId: 'bitbucket.service.user'
  ])

commonpipeline {
    nodes     = 'BUILD' /* label of jenkins nodes*/
    timeout   = '60'
}
1

I believe you are heading in the wrong direction. The way to implement what you want is to separate the "devops infrastructure logic" (placed in the 'ci-repo') from the pipeline trigger - a jenkinsfile inside the dev-repo, which includes the "devops infrastructure logic" as a tagged artifact.

Pros:

  • You have a central place to develop (and test) devops code
  • When changes are backward compatible, you can update all pipelines by moving the tag
  • When changes are NOT backward compatible, repos & specific branches can gradually move to a new tag
  • There is room for specific repos & branches to create their unique process if they have to

Cons:

  • Yes, sometimes people will need to update Jenkinsfiles in multiple places as they are accepting a new version. I actually think it's good.
0

There may not be a way to make your idea work with submodules. When a Jenkins multibranch pipeline polls SCM for changes, it has no visibility into submodules. I verified this recently. The result in "Scan Multibranch Pipeline Log" was a Communication error for url response. The Jenkinsfile was not found, and so no build was triggered.

The good news is that I discovered a straightforward way to achieve what we're after without using submodules. Have a look at the Remote Jenkinsfile Provider Plugin.

When a pipeline is polling a repository to trigger new builds, it looks for two things for each branch:

  1. The polled branch has new SCM changes since the last build (HEAD is different).
  2. A Jenkinsfile exists in repository X on branch Y at path Z.

In a multibranch pipeline's Build Configuration section, normally only one Mode is available to tell Jenkins where to find the Jenkinsfile: "by Jenkinsfile". This mode allows us to specify Z (path) only; X and Y are implicitly set to the repository and branch being polled.

What the Remote Jenkinsfile Provider Plugin gives us is a new Build Configuration Mode: "by Remote Jenkins File Plugin". This mode allows us to explicitly set the repository, branch, and path for the Jenkinsfile - X, Y, and Z. In your example, X would be ci-repo, but any Jenkinsfile in any repository is fair game. You could have ci-repo be a submodule of dev-repo if it's convenient, but no such relationship is required to make this work.

If desired, a "Match branches" option uses the polled branch name for Y, looking for a remote Jenkinsfile on a branch of the same name. For example, while polling dev-repo/feature-123, it would look for a Jenkinsfile on ci-repo/feature-123.

As long as the Jenkinsfile exists at the specified location, the second criteria for triggering new builds will be satisfied. You'll see something like this in the pipeline's polling logs:

Looking up DR/dev-repo for branches
Checking branch master from DR/dev-repo
No local file defined. Skipping Source Code SCM probe, since Jenkinsfile will be provided by Remote Jenkins File Plugin
    Met criteria
Scheduled build for branch: master

Instructions for how to set this up, with screenshots, can be found on the plugin's page.

Caveat: Pushing changes to a remote Jenkinsfile (ci-repo) will not trigger any builds of the polling repository (dev-repo). If this behavior is desired, one solution would be to use webhooks. Another would be to create a pipeline that polls the remote Jenkinsfile's repository and builds other pipeline jobs as desired using the build step, like so:

build(job: "dev-repo/${BRANCH_NAME}", wait: false)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.