Jenkins Artifactory plugin not downloading in multi-thread

Question

In my company we're having a bad time trying to download with the Artifactory plugin in Jenkins because we have a lot of tiny files in our repository and apparently the plugin works in a single thread, so it takes a REALLY long time. We tested the JFrog CLI where we can set the number of threads for the download and that actually works great. Still, we really need to use the original plugin.

Is there any workaround for this? Maybe it has something to do with the Artifactory server configuration?

Note: We have so many small files because that's how the development is being done in the company I'm afraid. We thought about archiving all the files before we store them but that would probably be non-economical in terms of storage, since every uploaded zip file would probably have a different checksum.

Answer 1

You could try to run the artifactory file upload in parallel if you are using the Jenkinsfile syntax: https://github.com/jenkinsci/pipeline-examples/blob/master/pipeline-examples/parallel-from-list/parallelFromList.groovy

Here is a simpler example to run things in parallel if you only have a fixed number of things you want to do in parallel:

parallel (
    "task1" : {
        //runTask1
    },
    "task2" : {
        //runTask2
    },
    "task3" : {
        //runTask3
    },
)

This can be enclosed inside a node, in which case they would all share the same workspace or each command can define node inside the task code to run into another node in which case it would have a different workspace for each task.