2

In declarative pipelines, Jenkins allows the definition of parallel stages. It further allows scripted pipeline general purpose scripts to create and manipulate the artifacts of the declarative pipeline. For example, it can create stages dynamically.

I would like to know how I can dynamically create parallel executing stages. Please note I am not talking about steps or actions, which are solved here:

How to properly achieve dynamic parallel action with a declarative pipeline?

Below is a simple Jenkinsfile that dynamically adds stages using groovy code. These stages are sequential. I would like a Jenkinsfile that adds stages dynamically as below, but uses the parallel construct at the stage level. As a result, three stages that run parallel should be generated by the program.

pipeline {
    agent any
    stages {
        stage('Add regression tests') {
            steps {
                addStage('offline','open-source','webgoat')
                addStage('offline','open-source','juice-shop')
                addStage('online','internal','our-website')
            }
        }
    }
}

final void addStage(final String type, final String suite,final String application) {
    script {
        stage("Run ${type} application  '\n${application}' in test suite '${suite}'") {

            runTestOnApplication(type,suite,application)

        }
    }
}

final void runTestOnApplication(final String type, final String suite,final String application) {
    labelledShell label: "Run test shell script for ${type} application '${application}' in test suite '${suite}'", script: 'test.bash'
}

  • Could you check whether one of the answers works for you? – 030 Dec 27 '19 at 16:12
  • +1 for this question. Fixing this is much more complicated to fix than it should be. – tftd Dec 27 '19 at 16:12
  • @tftd Do you know whether there is a related Jira Jenkins issue for this? If not, perhaps you could create it and link it to this Q&A. Perhaps creating an issue on the Jenkins Jira board could create awareness to make it less complicated? – 030 Dec 27 '19 at 16:25
  • 1
    @030 It looks like JENKINS-53032 asking for something similar. – tftd Dec 27 '19 at 16:34
  • 1
    @tftd Well done. Perhaps you could add a link to this ticket to your answer and a link to your answer to the jenkins ticket? – 030 Dec 27 '19 at 16:37
3
+50

This is quite complicated to achieve in Jenkins. We had a similar issue and here's how we solved it:

  1. We have a shared library where we keep our scripts ins ./vars/someScriptName.groovy
  2. We created a groovy script that generates the stages.

Here's a working pipeline code for you to try it out:

def generateITParallelStages(body)
{
    def config = [:]
    config.stages = []

    body.resolveStrategy = Closure.DELEGATE_FIRST
    body.delegate = config
    body()

    def buildStages = [:]
    config.stages.each { String stage, closure ->
        // default settings, if you want to have any
        def settings = [:]
        settings.type = 'online'; // if you don't define the type, it will be 'online' by default
        settings.suite = 'internal'; // if you don't define the suite it will be 'internal' by default

        closure.resolveStrategy = Closure.DELEGATE_FIRST
        closure.delegate = settings
        closure()

        buildStages.put(stage, prepareStage(stage, settings))
    }

    return buildStages
}

def prepareStage(String stage, Map settings)
{
    return {
        print "Run ${settings.type} application  '\\n${stage}' in test suite '${settings.suite}'"
    }
}

pipeline {
    agent any
    stages {
        stage('Add regression tests') {
            steps {
                script {
                    parallel generateITParallelStages {
                        stages = [
                                webgoat: {
                                    type = 'offline'
                                    suite = 'open-source'
                                },
                                'juice-shop': {
                                    type = 'offline'
                                    suite = 'open-source'
                                },
                                'our-website': {
                                    type = 'online'
                                    suite = 'internal'
                                }
                        ]
                    }
                }
            }
        }
    }
}

And here's the result:

pipeline-view

This is the only way we have been able to dynamically generate the parallel stages. Everything else will fail (at least at the time of writing this).

Other people seem to be having a hard time figuring this out as well - JENKINS-53032. If you feel the same make some noise :)

| improve this answer | |
  • 1
    I would like to set a docker agent on the stage, but I am getting an error that seems to indicate I can only use step interface classes as operations in the stage execution. Where would I define this setting? I have tried to find a JavaDoc that shows the underlying classes of the API, but have come up empty. – Jörn Guy Süß Jan 12 at 23:15
  • You would need to set the Add regression tests to run on none agent and then have in the prepareStage a node(settings.agent) { ... } which starts the parallel stage. – tftd Jan 12 at 23:19
  • 1
    Sorry for being such a nuisance. Is there a way I can add a sequence to the stage? I have tried calling the stages {} closure, but it is not valid. I assume I am in a scripted environment now, guessing from your previous instruction, which works great! – Jörn Guy Süß Jan 13 at 3:24
  • That's correct - while in prepareStage you are writing scripted pipeline code. This means, there is no post {}, no stages {} and other "sugar" from the declarative pipes. I assume you want to add a few stages on top of the running parallel stage (i.e. have unit-test, integration-test in juice-shop). To do that, just have node(settings.agent) { stage('ut') {...} stage('it') {...} }. Checkout jenkins.io/doc/book/pipeline/#scripted-pipeline-fundamentals. If that's not what you want - please open up a new question and I'll try to answer. – tftd Jan 13 at 13:44
1

I hope this helps. We had a similar issue and solved it using the below method.

Our project contains about 12 or so different components (programs) that we wanted to build in parallel. Because some take longer than others, we decided to break them up into 4 individual "tracks" that would run in parallel. To begin, assume we have an array where each element is a new component object. The object has some properties, like name for the name of the component and track for which track, 1, 2, 3, or 4, that the component is assigned to. I'm not focusing on this part as it's probably not relevant to what you're trying to do.

First we define our createParallelStage function which iterates over an ArrayList of components, setting them up to build.

def createParallelStage(ArrayList components) {
    return{
        // iterate over list of components objects
        components.each { component ->
            dir(buildFolder) {
                // build the component
                sh "./build.sh ${component}"

            }
        }
    }
}

We'll use a list of component objects called BuildComponents, iterate over them, and add them to parallelGroups[i] where i is the value of the objects track property

BuildComponents.each { component ->
    parallelGroups[component.track].add(component.name)
    echo "${component.name} is going to parallel track ${component.track}"
    // example: IPCService is going to parallel track 3
    }
}

We're now left with iterating over parallelGroups, passing each element to createParallelStage() while adding it to the parallelStages ArrayList.

parallelGroups.each{i, component ->
    parallelStages.put(i, createParallelStage(component))
}

We then set failFast to true, so if any of our components fail, the whole build job will fail, and then pass our list of parallelStages to the parallel() function.

parallelStages.failFast = true
parallel(parallelStages)

I hope this was specific to the question you were asking. I wish I could be a little more detailed, but then it might be too specific of a solution to work for you.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.