Triggering specific pipeline builds for monorepos in Jenkins

Question

I'm in the process of converting multiple repositories into a single repository, our CI tool of choice is Jenkins due to the conversion of multiple repository structures into a single one 2 main issues have arisen.

1. Build/test times have increased significantly as all the builds/tests have to be run for every single commit. This is partially alleviated by using a build tool, in our case we have gone with using Buck.

2. After all the tests associated with the committed code are run, I have a deployment Jenkinsfile for each project. How will I be able to only trigger the Jenkinsfiles for projects that need to be re-deployed? And if I am able to do so, is this a correct practice?

Answer 1

1. Get a list of changed files and use that to determine which tests to run.

2. Load external Groovy scripts at run time in order to run your deploys.

Answer 2

You can use the "when" block combined with the built in "changeset" condition to conditionally run only certain stages of your monorepo's pipeline. From the when.changeset documentation:

changeset- Executes the stage if the build’s SCM changeset contains one or more files matching the given string or glob. Example: when { changeset "**/*.js" }

Here is an example Jenkinsfile using this strategy:

pipeline {
    agent any
    stages {
        stage('build matchengine') {
            when {
                changeset "**/matchengine/*.*"
            }
            steps {
                echo 'building match engine'
            }
        }
        stage('build posttrade') {
            when {
                changeset "**/posttrade/*.*"
            }
            steps {
                echo 'building post trade'
            }
        }
    }
}

, applicable to the monorepo project structure shown below:

 .(my-project)
   |-- Jenkinsfile
   |-- matchengine
   |-- posttrade
   |-- serverless
   |-- ui

This strategy won't scale past small codebases because it would be hard to keep track of which modules depend one each other. You would be better off using a build system like Bazel. Your CI job would simply issue a bazel build //... (build everything), and Bazel would calculate what actually needs to be built, and what needs to be tested. Further, there even exist bazel rules such as rules_docker and rules_k8s which can calculate which of your containers need to be rebuilt and pushed to a container registry, and which of your applications need to be redeployed to Kubernetes.