How to define dynamic parallel stages in a Jenkinsfile?

Question

In declarative pipelines, Jenkins allows the definition of parallel stages. It further allows scripted pipeline general purpose scripts to create and manipulate the artifacts of the declarative pipeline. For example, it can create stages dynamically.

I would like to know how I can dynamically create parallel executing stages. Please note I am not talking about steps or actions, which are solved here:

How to properly achieve dynamic parallel action with a declarative pipeline?

Below is a simple Jenkinsfile that dynamically adds stages using groovy code. These stages are sequential. I would like a Jenkinsfile that adds stages dynamically as below, but uses the parallel construct at the stage level. As a result, three stages that run parallel should be generated by the program.

pipeline {
    agent any
    stages {
        stage('Add regression tests') {
            steps {
                addStage('offline','open-source','webgoat')
                addStage('offline','open-source','juice-shop')
                addStage('online','internal','our-website')
            }
        }
    }
}

final void addStage(final String type, final String suite,final String application) {
    script {
        stage("Run ${type} application  '\n${application}' in test suite '${suite}'") {

            runTestOnApplication(type,suite,application)

        }
    }
}

final void runTestOnApplication(final String type, final String suite,final String application) {
    labelledShell label: "Run test shell script for ${type} application '${application}' in test suite '${suite}'", script: 'test.bash'
}

Answer 1

This is quite complicated to achieve in Jenkins. We had a similar issue and here's how we solved it:

1. We have a shared library where we keep our scripts ins ./vars/someScriptName.groovy
2. We created a groovy script that generates the stages.

Here's a working pipeline code for you to try it out:

def generateITParallelStages(body)
{
    def config = [:]
    config.stages = []

    body.resolveStrategy = Closure.DELEGATE_FIRST
    body.delegate = config
    body()

    def buildStages = [:]
    config.stages.each { String stage, closure ->
        // default settings, if you want to have any
        def settings = [:]
        settings.type = 'online'; // if you don't define the type, it will be 'online' by default
        settings.suite = 'internal'; // if you don't define the suite it will be 'internal' by default

        closure.resolveStrategy = Closure.DELEGATE_FIRST
        closure.delegate = settings
        closure()

        buildStages.put(stage, prepareStage(stage, settings))
    }

    return buildStages
}

def prepareStage(String stage, Map settings)
{
    return {
        print "Run ${settings.type} application  '\\n${stage}' in test suite '${settings.suite}'"
    }
}

pipeline {
    agent any
    stages {
        stage('Add regression tests') {
            steps {
                script {
                    parallel generateITParallelStages {
                        stages = [
                                webgoat: {
                                    type = 'offline'
                                    suite = 'open-source'
                                },
                                'juice-shop': {
                                    type = 'offline'
                                    suite = 'open-source'
                                },
                                'our-website': {
                                    type = 'online'
                                    suite = 'internal'
                                }
                        ]
                    }
                }
            }
        }
    }
}

And here's the result:

This is the only way we have been able to dynamically generate the parallel stages. Everything else will fail (at least at the time of writing this).

Other people seem to be having a hard time figuring this out as well - JENKINS-53032. If you feel the same make some noise :)

Answer 2

I hope this helps. We had a similar issue and solved it using the below method.

Our project contains about 12 or so different components (programs) that we wanted to build in parallel. Because some take longer than others, we decided to break them up into 4 individual "tracks" that would run in parallel. To begin, assume we have an array where each element is a new component object. The object has some properties, like name for the name of the component and track for which track, 1, 2, 3, or 4, that the component is assigned to. I'm not focusing on this part as it's probably not relevant to what you're trying to do.

First we define our createParallelStage function which iterates over an ArrayList of components, setting them up to build.

def createParallelStage(ArrayList components) {
    return{
        // iterate over list of components objects
        components.each { component ->
            dir(buildFolder) {
                // build the component
                sh "./build.sh ${component}"

            }
        }
    }
}

We'll use a list of component objects called BuildComponents, iterate over them, and add them to parallelGroups[i] where i is the value of the objects track property

BuildComponents.each { component ->
    parallelGroups[component.track].add(component.name)
    echo "${component.name} is going to parallel track ${component.track}"
    // example: IPCService is going to parallel track 3
    }
}

We're now left with iterating over parallelGroups, passing each element to createParallelStage() while adding it to the parallelStages ArrayList.

parallelGroups.each{i, component ->
    parallelStages.put(i, createParallelStage(component))
}

We then set failFast to true, so if any of our components fail, the whole build job will fail, and then pass our list of parallelStages to the parallel() function.

parallelStages.failFast = true
parallel(parallelStages)

I hope this was specific to the question you were asking. I wish I could be a little more detailed, but then it might be too specific of a solution to work for you.