3

Introduction

We have a multibranch pipeline project. The build configuration is defined by a jenkinsfile. The system consists of one master and a couple of agents.

The stages are run in parallel, and for each one we do a checkout of the repository.

See attached file below.

The Problem

is that the size of the workspace accumulates greatly as the amount of branches increases.

The Solution #1

Periodic cleanup of the workspace. A custom script will be written. Something like this one:

// Check if a slave has < 10 GB of free space, wipe out workspaces if it does

import hudson.model.*;
import hudson.util.*;
import jenkins.model.*;
import hudson.FilePath.FileCallable;
import hudson.slaves.OfflineCause;
import hudson.node_monitors.*;

for (node in Jenkins.instance.nodes) {
    computer = node.toComputer()
    if (computer.getChannel() == null) continue

    rootPath = node.getRootPath()
    size = DiskSpaceMonitor.DESCRIPTOR.get(computer).size
    roundedSize = size / (1024 * 1024 * 1024) as int

    println("node: " + node.getDisplayName() + ", free space: " + roundedSize + "GB")
    if (roundedSize < 10) {
        computer.setTemporarilyOffline(true, new hudson.slaves.OfflineCause.ByCLI("disk cleanup"))
        for (item in Jenkins.instance.items) {
            jobName = item.getFullDisplayName()

            if (item.isBuilding()) {
                println(".. job " + jobName + " is currently running, skipped")
                continue
            }

            println(".. wiping out workspaces of job " + jobName)

            workspacePath = node.getWorkspaceFor(item)
            if (workspacePath == null) {
                println(".... could not get workspace path")
                continue
            }

            println(".... workspace = " + workspacePath)

            customWorkspace = item.getCustomWorkspace()
            if (customWorkspace != null) {
                workspacePath = node.getRootPath().child(customWorkspace)
                println(".... custom workspace = " + workspacePath)
            }

            pathAsString = workspacePath.getRemote()
            if (workspacePath.exists()) {
                workspacePath.deleteRecursive()
                println(".... deleted from location " + pathAsString)
            } else {
                println(".... nothing to delete at " + pathAsString)
            }
        }

        computer.setTemporarilyOffline(false, null)
    }
}

## The Solution #2 ##
One checkout per job and node. Thus each parallel stage will execute one executable instance of the same repository. For me personaly this is the preferred method, although more complex than solution #1.

pseudocode:

    stage("Build") {
        try {
            parallel builds
        } catch (err) {
        }
    }

    def build(predef) {
        return {
            node() {
                checkout_repository(${env.NODE_NAME})
                Execute build
                ...
            }
        }
    }

    def checkout_repository(node){
        No checkout for this node
            Flag that we are checking out a repository on node (think ~semaphore)
            Cheking out ..
            Flag that repository exist on node
    }

The Question

  1. What are the cons in solution #2
  2. Does it follow the jenkins way of doing it. Or this to much freestyle?

The job: The Job

  • Another disadvantage of long-lived branches - one of the CI Theatre patterns... – Dan Cornilescu Feb 13 '18 at 14:03
  • 1
    For #2 to work correctly you need your build structure to be pristine - it's not uncommon for less-than-perfect builds running in parallel to interfere with each-other: race conditions creating/deleting directories, overwriting artifacts, too high peaks of resource usage, etc - typically hard to repro intermittent failures. Just be aware of the possibility. – Dan Cornilescu Feb 13 '18 at 14:13
  • @DanCornilescu Interesting article. Thanks for sharing. You might want to put your comment as an answer. The possibility for intermittent failures when going for #2 is something I will take into consideration. – ezFreak Feb 14 '18 at 10:13
  • Can you detail a bit the multi-branch aspect? Do you mean different repositories, each with their own branches or different branches of the same repositories (and, if so, what's the branching stragey)? – Dan Cornilescu Feb 14 '18 at 12:38
  • The workflow is that for each story (task) we create a branch. This branch will then be picked up by Jenkins and run different tests. When story is done, we merge the branch back to trunk. There is only one repository (trunk), and we create branches from trunk. But the main issue I feel that we have is that for one job, we do one checkout per parallel stage. So if you look at the attached picture above, you can see that there is more than 9 parallel stages. So +9 checkouts per branch. – ezFreak Feb 14 '18 at 12:41
1

Your difficulty stems from encouraging potentially long-lived branches, which is deviating from the CI methodology - which is supposedly why you'd be using jenkins - a CI tool - in the first place.

Normally you shouldn't be running the tests for those branches in the Jenkins pipeline for the trunk - you want to measure the quality of the integrated work, not that of the work done in the silos.

I do understand the desire to run such tests before merging into trunk, so maybe it's OK to use jenkins to do it if you want to, but with separate jobs/pipelines, not tied into the trunk pipeline. And keep in mind that the results of such tests aren't necessarily conclusive: just because they pass over there doesn't mean the same tests will also pass in the trunk, you'll need very strict conditions to make that always true.

Personally I'd go for a fully automated patcing/merging solution based on pre-commit verifications which can enforce the strict conditions necessary to prevent post-merge trunk quality regressions (but I'm biased, I built such solutions). See for example How to ensure that git subtrees are kept up to date?

Anyways, back to the question. Since the code for the various pipeline segments that you seek to perform in parallel comes from different branches you won't be able to use the same workspace, you have to pull separate workspaces from the respective, different branches. Which makes the question kinda moot.

Final note: assuming the same workspace could be used in multiple builds, your #2 approach requires a pristine build structure. It's not uncommon for less-than-perfect builds running in parallel in the same workspace to interfere with each-other: race conditions creating/deleting directories, overwriting artifacts, too high peaks of resource usage, etc - typically hard to repro intermittent failures. These could easily happen if, for example, the builds would be for the same product but for different CPU architectures, or builds including code shared across multiple products. Just be aware of the possibility.

  • Excellent input. I believe this information can be useful for others as well. Satisfied with your answer and thus accepting it. – ezFreak Feb 15 '18 at 9:09
  • +1 for "Personally I'd go for a fully automated patcing/merging solution based on pre-commit verifications which can enforce the strict conditions necessary to prevent post-merge trunk quality regressions (but I'm biased, I built such solutions). See for example How to ensure that git subtrees are kept up to date?" – ezFreak Feb 15 '18 at 9:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.