One Checkout When Running Parallel Stages In Jenkins

Question

Introduction

We have a multibranch pipeline project. The build configuration is defined by a jenkinsfile. The system consists of one master and a couple of agents.

The stages are run in parallel, and for each one we do a checkout of the repository.

See attached file below.

The Problem

is that the size of the workspace accumulates greatly as the amount of branches increases.

The Solution #1

Periodic cleanup of the workspace. A custom script will be written. Something like this one:

// Check if a slave has < 10 GB of free space, wipe out workspaces if it does

import hudson.model.*;
import hudson.util.*;
import jenkins.model.*;
import hudson.FilePath.FileCallable;
import hudson.slaves.OfflineCause;
import hudson.node_monitors.*;

for (node in Jenkins.instance.nodes) {
    computer = node.toComputer()
    if (computer.getChannel() == null) continue

    rootPath = node.getRootPath()
    size = DiskSpaceMonitor.DESCRIPTOR.get(computer).size
    roundedSize = size / (1024 * 1024 * 1024) as int

    println("node: " + node.getDisplayName() + ", free space: " + roundedSize + "GB")
    if (roundedSize < 10) {
        computer.setTemporarilyOffline(true, new hudson.slaves.OfflineCause.ByCLI("disk cleanup"))
        for (item in Jenkins.instance.items) {
            jobName = item.getFullDisplayName()

            if (item.isBuilding()) {
                println(".. job " + jobName + " is currently running, skipped")
                continue
            }

            println(".. wiping out workspaces of job " + jobName)

            workspacePath = node.getWorkspaceFor(item)
            if (workspacePath == null) {
                println(".... could not get workspace path")
                continue
            }

            println(".... workspace = " + workspacePath)

            customWorkspace = item.getCustomWorkspace()
            if (customWorkspace != null) {
                workspacePath = node.getRootPath().child(customWorkspace)
                println(".... custom workspace = " + workspacePath)
            }

            pathAsString = workspacePath.getRemote()
            if (workspacePath.exists()) {
                workspacePath.deleteRecursive()
                println(".... deleted from location " + pathAsString)
            } else {
                println(".... nothing to delete at " + pathAsString)
            }
        }

        computer.setTemporarilyOffline(false, null)
    }
}

## The Solution #2 ##
One checkout per job and node. Thus each parallel stage will execute one executable instance of the same repository. For me personaly this is the preferred method, although more complex than solution #1.

pseudocode:

    stage("Build") {
        try {
            parallel builds
        } catch (err) {
        }
    }

    def build(predef) {
        return {
            node() {
                checkout_repository(${env.NODE_NAME})
                Execute build
                ...
            }
        }
    }

    def checkout_repository(node){
        No checkout for this node
            Flag that we are checking out a repository on node (think ~semaphore)
            Cheking out ..
            Flag that repository exist on node
    }

The Question

1. What are the cons in solution #2
2. Does it follow the jenkins way of doing it. Or this to much freestyle?

The job:

Answer 1

Your difficulty stems from encouraging potentially long-lived branches, which is deviating from the CI methodology - which is supposedly why you'd be using jenkins - a CI tool - in the first place.

Normally you shouldn't be running the tests for those branches in the Jenkins pipeline for the trunk - you want to measure the quality of the integrated work, not that of the work done in the silos.

I do understand the desire to run such tests before merging into trunk, so maybe it's OK to use jenkins to do it if you want to, but with separate jobs/pipelines, not tied into the trunk pipeline. And keep in mind that the results of such tests aren't necessarily conclusive: just because they pass over there doesn't mean the same tests will also pass in the trunk, you'll need very strict conditions to make that always true.

Personally I'd go for a fully automated patcing/merging solution based on pre-commit verifications which can enforce the strict conditions necessary to prevent post-merge trunk quality regressions (but I'm biased, I built such solutions). See for example How to ensure that git subtrees are kept up to date?

Anyways, back to the question. Since the code for the various pipeline segments that you seek to perform in parallel comes from different branches you won't be able to use the same workspace, you have to pull separate workspaces from the respective, different branches. Which makes the question kinda moot.

Final note: assuming the same workspace could be used in multiple builds, your #2 approach requires a pristine build structure. It's not uncommon for less-than-perfect builds running in parallel in the same workspace to interfere with each-other: race conditions creating/deleting directories, overwriting artifacts, too high peaks of resource usage, etc - typically hard to repro intermittent failures. These could easily happen if, for example, the builds would be for the same product but for different CPU architectures, or builds including code shared across multiple products. Just be aware of the possibility.