1

I've been using a Jenkins pipeline with a parallel step for testing like this:

pipeline {
    agent { label 'php' }
    stages {
        stage('build') {
            steps {
                ...
            }
        }
        stage('test') {
            parallel {
                stage('unit tests') {
                    steps {
                        ...
                    }
                }
                stage('integration tests') {
                    steps {
                        ...
                    }
                }
            }
        }
        stage('deploy') {
            steps {
                ...
            }
        }
    }
}

The pipeline runs fine on a single agent, but now I want to add a step with manual input, which should not keep the agent running. The agents are cloud servers that get started and stopped on demand.

To do so, I assigned "none" agent to the pipeline and the actual agent to all stages but the one requesting input:

pipeline {
    agent none
    stages {
        stage('build') {
            agent { label 'php' }
            steps {
                ...
            }
        }
        stage('test') {
            parallel {
                stage('unit tests') {
                    agent { label 'php' }
                    steps {
                        ...
                    }
                }
                stage('integration tests') {
                    agent { label 'php' }
                    steps {
                        ...
                    }
                }
            }
        }
        stage('deploy staging') {
            agent { label 'php' }
            steps {
                ...
            }
        }
        stage('confirm release') {
            steps {
                input { message 'release?' }
            }
        }
        stage('deploy production') {
            agent { label 'php' }
            steps {
                ...
            }
        }
    }
}

Artifacts that are needed in a following stage are carried over with stash and unstash. This works, but with two nuisances:

  1. Stashing and unstashing between each single stage causes overhead. I would rather have all stages until the manual input executed in a single workspace. But agent is only allowed in stages that contain steps.
  2. Each parallel stage blocks a single agent. So if there is only one free agent, there is no parallel execution, unlike before.

I can live with (1) if it's not possible otherwise, but parallel execution on one agent is important for me. Is it possible somehow in my current setup? If not, what are my options to achieve something similar?

1 Answer 1

1

I modified your Jenkins file as shown below.

pipeline {
agent { label 'agent01'}
stages {
    stage('build') {
        steps {
            echo "build"
            sh "ls -lrt && touch build.txt"
        }
    }
    stage('test') {
        parallel {
            stage('unit tests') {
                steps {
                    echo "unit tests"
                    sh "ls -lrt && touch ut.txt"
                    sleep(time:10,unit:"SECONDS")
                }
            }
            stage('integration tests') {
                steps {
                    echo "integration tests"
                    sh "ls -lrt && touch it.txt"
                    sleep(time:10,unit:"SECONDS")
                }
            }
        }
    }
    stage('deploy staging') {
        agent { label 'master' }
        steps {
            echo "deploy staging"
            sh "ls -lrt && touch dep.txt"
        }
    }
    stage('confirm release') {
        agent { label 'master' }
        steps {
            echo "RELEASE?"
            sh "ls -lrt && touch release.txt"
        }
    }
    stage('deploy production') {
        steps {
            echo "deploy production"
            sh "ls -lrt && touch produ.txt"
        }
    }
}
}
  1. You need not stash and unstash files since it runs on same node and workspace.
  2. Use a general node for all the stages except for the one that needs a user input.

The output is

    Running in Durability level: MAX_SURVIVABILITY
[Pipeline] node
Running on agent01 in /var/jenkins_home/workspace/sandbox-pipeline
[Pipeline] {
[Pipeline] stage
[Pipeline] { (build)
[Pipeline] echo
build
[Pipeline] sh
[sandbox-pipeline] Running shell script
+ ls -lrt
total 0
+ touch build.txt
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (test)
[Pipeline] parallel
[Pipeline]     [unit tests] { (Branch: unit tests)
[Pipeline]     [integration tests] { (Branch: integration tests)
[Pipeline]     [unit tests] stage
[Pipeline]     [unit tests] { (unit tests)
[Pipeline]     [integration tests] stage
[Pipeline]     [integration tests] { (integration tests)
[Pipeline]     [unit tests] echo
[unit tests] unit tests
[Pipeline]     [unit tests] sh
[unit tests]     [sandbox-pipeline] Running shell script
[Pipeline]     [integration tests] echo
[integration tests] integration tests
[Pipeline]     [integration tests] sh
[unit tests] + ls -lrt
[unit tests] total 0
[unit tests] -rw-r--r-- 1 jenkins docker 0 Oct 19 13:53 build.txt
[unit tests] + touch ut.txt
[integration tests]     [sandbox-pipeline] Running shell script
[Pipeline]     [unit tests] sleep
[integration tests] + ls -lrt
[integration tests] total 0
[integration tests] -rw-r--r-- 1 jenkins docker 0 Oct 19 13:53 build.txt
[integration tests] -rw-r--r-- 1 jenkins docker 0 Oct 19 13:53 ut.txt
[integration tests] + touch it.txt
[unit tests] Sleeping for 10 sec
[Pipeline]     [integration tests] sleep
[integration tests] Sleeping for 10 sec
[Pipeline]     [unit tests] }
[Pipeline]     [unit tests] // stage
[Pipeline]     [unit tests] }
[Pipeline]     [integration tests] }
[Pipeline]     [integration tests] // stage
[Pipeline]     [integration tests] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy staging)
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/sandbox-pipeline
[Pipeline] {
[Pipeline] echo
deploy staging
[Pipeline] sh
[sandbox-pipeline] Running shell script
+ ls -lrt
total 0
+ touch dep.txt
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (confirm release)
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/sandbox-pipeline
[Pipeline] {
[Pipeline] echo
RELEASE?
[Pipeline] sh
[sandbox-pipeline] Running shell script
+ ls -lrt
total 0
-rw-r--r-- 1 jenkins 115 0 Oct 19 13:58 dep.txt
+ touch release.txt
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy production)
[Pipeline] echo
deploy production
[Pipeline] sh
[sandbox-pipeline] Running shell script
+ ls -lrt
total 0
-rw-r--r-- 1 jenkins docker 0 Oct 19 13:53 build.txt
-rw-r--r-- 1 jenkins docker 0 Oct 19 13:53 ut.txt
-rw-r--r-- 1 jenkins docker 0 Oct 19 13:53 it.txt
+ touch produ.txt
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
2
  • Is this actually valid? I haven't tried it, but my impression is that you can't define an agent in a child scope once one has been defined in a parent. The exception being that you can target specific agents using node(...) inside a script { ... } block. Can anyone clarify? Jun 8, 2021 at 15:10
  • 1
    @MaxCascone The code I provided above is tested and the log also confirms that you can choose different agents. Please note that it is a declarative pipeline.
    – Ram
    Jun 8, 2021 at 16:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.