33

Currently, I'm going to need an implementation that must find all files within a directory and start a parallel task for every file found.

Is it possible to achieve this using declarative pipelines?

pipeline {
    agent any
    stages {
        stage("test") {
            steps {
                dir ("file_path") {
                    // find all files with complete path
                    parallel (
                        // execute parallel tasks for each file found.
                        // this must be dynamic
                        }
                    }
                }
            }
        }
    }
}
Improve this question
2
  • How can I do if i want to run multiple steps sequentaly and not parallel?
    – Frank Escobar
    Commented Feb 27, 2019 at 18:06
  • Sure, but that way you cannot generate the parallel tasks dynamically, for example, depending on some files on the repository.
    – Raúl Salinas-Monteagudo
    Commented Jan 29, 2020 at 12:00

5 Answers 5

Reset to default
40

Managed to solve it with the following code:

pipeline {
    agent { label "master"}
    stages {
        stage('1') {
            steps {
                script {
                    def tests = [:]
                    for (f in findFiles(glob: '**/html/*.html')) {
                        tests["${f}"] = {
                            node {
                                stage("${f}") {
                                    echo '${f}'
                                }
                            }
                        }
                    }
                    parallel tests
                }
            }
        }       
    }
}
Improve this answer
6
  • Please also check out the official Pipeline examples - jenkins.io/doc/pipeline/examples/#parallel-multiple-nodes
    – phedoreanu
    Commented Apr 10, 2018 at 16:15
  • @phedoreanu i'm using declarative pipeline...
    – thclpr
    Commented Apr 10, 2018 at 16:24
  • @phedoreanu I have rejected your edit, editing code should have good reasons, your comment isn't enough for me to allow an edit of this kind on an answer which was a self solution. I think you should have commented to discuss the matter with the answer author before doing this edit.
    – Tensibai
    Commented Apr 16, 2018 at 9:38
  • 1
    @phedoreanu I you think you have a better derivative work, then please write your own answer and explain why it is better (in error handling, templates, etc.) instead.
    – Tensibai
    Commented Apr 16, 2018 at 9:40
  • Hi, I figured out the same after a few failing attempts. My only problem right now is that if I put two stage{..} sections inside a node for some reasons the workflow stage chart and Blu Ocean get confused. For instance in the workflow stage chart I get NaNy NaNd and in Blue Ocean I get only the first stage.
    – Giuseppe Salvatore
    Commented Apr 11, 2019 at 10:13
14

This also works, if you want to stay within the Declarative Pipeline space

// declare our vars outside the pipeline
def tests = [:]
def files

pipeline {
    agent any
    stages {
        stage('1') {
            steps {
                script {
                    // we've declared the variable, now we give it the values
                    files = findFiles(glob: '**/html/*.html')
                    // Loop through them
                    files.each { f ->
                        // add each object from the 'files' loop to the 'tests' array
                        tests[f] = {
                            // we're already in the script{} block, so do our advanced stuff here
                            echo f.toString()
                        }
                    }
                    // Still within the 'Script' block, run the parallel array object
                    parallel tests
                }
            }
        }       
    }
}
Improve this answer
1
  • 1
    If you want to allocate each parallel task to different Jenkins nodes then simply wrap the actions in a node {} block, like this: tests[f] = { node { echo f.toString() } }
    – primetheus
    Commented Jan 29, 2019 at 19:57
7

Be aware, that dynamic build steps could cause some problems in some build steps, e.g. when you call an other job:

pipeline {
    stages {
        stage('Test') {
            steps {
                script {
                    def tests = [:]
                    for (f in findFiles(glob: '**/html/*.html')) {
                        // Create temp variable, otherwise the name will be the last value of the for loop
                        def name = f
                        tests["${name}"] = {
                            build job: "${name}"
                        }
                    }
                    parallel tests
                }
            }
        }       
    }
}
Improve this answer
1
  • I don't know why but i used stage and echo inside tests, but it will giving stage name and echo as null until I used a local variable like you have used name above. Do you know why that is? Thanks for your answer though!
    – Kaushik
    Commented Apr 8, 2020 at 10:26
2

It's much easier to use scripted Pipelines to do this since you can use arbitrary Groovy, but you should still be able to do this with declarative Pipelines using the findFiles step.

Improve this answer
0
0

Here is a solution where you can setup the maximum number of concurrent tasks running at once with collate(n).

pipeline {
  stages {
    stage("Parallel") {
      steps {
        script {
          def list = ["alpha", "beta", "gamma", "delta", "epsilon", "zeta", "eta", "theta", "iota", "kappa"]

          // Split the list into batches of 4
          def parallelBatches = list.collate(4);
          // [["alpha", "beta", "gamma", "delta"], ["epsilon", "zeta", "eta", "theta"], ["iota", "kappa"]]

          // This allows to run a maximum of 4 tasks in parallel (at once)
          for (parallelBatch in parallelBatches) {
            def tasks = [:]

            for (task in parallelBatch) {
              tasks["${task}"] = {
                stage("${task}") {
                  echo '${task}'
                }
              }
            }
            parallel tasks

            // Once the batch is over, the outer loop move onto the next batch
          }
        }
      }
    }
  }
}
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.