8

In my company we're having a bad time trying to download with the Artifactory plugin in Jenkins because we have a lot of tiny files in our repository and apparently the plugin works in a single thread, so it takes a REALLY long time. We tested the JFrog CLI where we can set the number of threads for the download and that actually works great. Still, we really need to use the original plugin.

Is there any workaround for this? Maybe it has something to do with the Artifactory server configuration?

Note: We have so many small files because that's how the development is being done in the company I'm afraid. We thought about archiving all the files before we store them but that would probably be non-economical in terms of storage, since every uploaded zip file would probably have a different checksum.

  • Typically one would package the application prior to uploading to artifactory. I was just wondering how was artifactory used in your development process, what is the platform or language used and how are these files consumed. Having a broader picture of the process could help us find potential solutions that might not be obvious when focusing on the download issue alone. – Newtopian May 10 '17 at 13:19
5

You could try to run the artifactory file upload in parallel if you are using the Jenkinsfile syntax: https://github.com/jenkinsci/pipeline-examples/blob/master/pipeline-examples/parallel-from-list/parallelFromList.groovy

Here is a simpler example to run things in parallel if you only have a fixed number of things you want to do in parallel:

parallel (
    "task1" : {
        //runTask1
    },
    "task2" : {
        //runTask2
    },
    "task3" : {
        //runTask3
    },
)

This can be enclosed inside a node, in which case they would all share the same workspace or each command can define node inside the task code to run into another node in which case it would have a different workspace for each task.

  • Hi Michael! I was talking more about downloading files and not uploading. If I did this with downloading, wouldn't it save the files on different nodes rather than a single workspace? – Asaf Haim May 5 '17 at 7:40
  • Not necessarily, I will give you a better example in a few moments – Michael Pereira May 5 '17 at 13:03
  • Not sure that matches my case since when I download artifacts, I don't know the exact number of artifacts (using ant-like syntax to download the entire repository) – Asaf Haim May 10 '17 at 12:38
  • In that case, take a look at jenkins.io/doc/pipeline/examples/#jobs-in-parallel You can reuse the same code and just change the loop counter to the number of elements you have to download, and replace the build job line with the one to download from artifactory – Michael Pereira May 10 '17 at 15:25
  • But what if I don't know the number of files I download? I use the double asterisk notation (**) to recursively download all my repository, so neither I know how many there are nor their names. – Asaf Haim May 14 '17 at 9:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.