25

Currently, I'm going to need an implementation that must find all files within a directory and start a parallel task for every file found.

Is it possible to achieve this using declarative pipelines?

pipeline {
    agent any
    stages {
        stage("test") {
            steps {
                dir ("file_path") {
                    // find all files with complete path
                    parallel (
                        // execute parallel tasks for each file found.
                        // this must be dynamic
                        }
                    }
                }
            }
        }
    }
}
  • How can I do if i want to run multiple steps sequentaly and not parallel? – Frank Escobar Feb 27 '19 at 18:06
  • Sure, but that way you cannot generate the parallel tasks dynamically, for example, depending on some files on the repository. – Raúl Salinas-Monteagudo Jan 29 at 12:00
31

Managed to solve it with the following code:

pipeline {
    agent { label "master"}
    stages {
        stage('1') {
            steps {
                script {
                    def tests = [:]
                    for (f in findFiles(glob: '**/html/*.html')) {
                        tests["${f}"] = {
                            node {
                                stage("${f}") {
                                    echo '${f}'
                                }
                            }
                        }
                    }
                    parallel tests
                }
            }
        }       
    }
}
| improve this answer | |
  • Please also check out the official Pipeline examples - jenkins.io/doc/pipeline/examples/#parallel-multiple-nodes – phedoreanu Apr 10 '18 at 16:15
  • @phedoreanu i'm using declarative pipeline... – thclpr Apr 10 '18 at 16:24
  • @phedoreanu I have rejected your edit, editing code should have good reasons, your comment isn't enough for me to allow an edit of this kind on an answer which was a self solution. I think you should have commented to discuss the matter with the answer author before doing this edit. – Tensibai Apr 16 '18 at 9:38
  • @phedoreanu I you think you have a better derivative work, then please write your own answer and explain why it is better (in error handling, templates, etc.) instead. – Tensibai Apr 16 '18 at 9:40
  • Hi, I figured out the same after a few failing attempts. My only problem right now is that if I put two stage{..} sections inside a node for some reasons the workflow stage chart and Blu Ocean get confused. For instance in the workflow stage chart I get NaNy NaNd and in Blue Ocean I get only the first stage. – Giuseppe Apr 11 '19 at 10:13
11

This also works, if you want to stay within the Declarative Pipeline space

// declare our vars outside the pipeline
def tests = [:]
def files

pipeline {
    agent any
    stages {
        stage('1') {
            steps {
                script {
                    // we've declared the variable, now we give it the values
                    files = findFiles(glob: '**/html/*.html')
                    // Loop through them
                    files.each { f ->
                        // add each object from the 'files' loop to the 'tests' array
                        tests[f] = {
                            // we're already in the script{} block, so do our advanced stuff here
                            echo f.toString()
                        }
                    }
                    // Still within the 'Script' block, run the parallel array object
                    parallel tests
                }
            }
        }       
    }
}
| improve this answer | |
  • 1
    If you want to allocate each parallel task to different Jenkins nodes then simply wrap the actions in a node {} block, like this: tests[f] = { node { echo f.toString() } } – primetheus Jan 29 '19 at 19:57
3

Be aware, that dynamic build steps could cause some problems in some build steps, e.g. when you call an other job:

pipeline {
    stages {
        stage('Test') {
            steps {
                script {
                    def tests = [:]
                    for (f in findFiles(glob: '**/html/*.html')) {
                        // Create temp variable, otherwise the name will be the last value of the for loop
                        def name = f
                        tests["${name}"] = {
                            build job: "${name}"
                        }
                    }
                    parallel tests
                }
            }
        }       
    }
}
| improve this answer | |
  • I don't know why but i used stage and echo inside tests, but it will giving stage name and echo as null until I used a local variable like you have used name above. Do you know why that is? Thanks for your answer though! – Kaushik Apr 8 at 10:26
1

It's much easier to use scripted Pipelines to do this since you can use arbitrary Groovy, but you should still be able to do this with declarative Pipelines using the findFiles step.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.