2

Although our own sources are checked out from a revision-control systems, the third-party software we depend on needs to be downloaded from their respective sites.

The plugin I'm looking for would offer a "Source Management" (or "Pre-Build") step, where I could register one (or more) sources for the same file. For example:

and its digest (such as SHA256).

At build- (or at a prebuild-) time, it would check its cache and, if the file is not present there, download and verify it, and make it available to the subsequent build-steps.

I'm sure something like this, or very similar, exists already, but I cannot find it. Sonatype's Nexus seems to offer this, but only for files already found in one's Nexus repository, not for arbitrary URL's.

Any help?

1 Answer 1

1

Perhaps you are looking for the Job Cacher plugin?

To quote the plugin page:

Summary

Enables caching of files on transient executors for use on subsequent builds to improve performance

Features

  • Item storage extension point supporting on master storage and AWS S3
  • Cache Wrapper for free style jobs that manages the cache Arbitrary File Cache where user specifies paths to be cached
  • UI on Job page to review the job's caches
  • Supports Pipeline jobs with a cache directive
  • Cache Extension Point for other plugins to provide opinionated caching capability such as Gradle caches
  • Incremental storing of cache into storage to minimize data movement

However, these are for files created by the jobs, not needed by them. In order to use it in your case, you might need a few dummy jobs specifically for downloading the content from the arbitrary URLs, and then caching them in your Jenkins cache. I guess you'd have to write the script for that.

To be frank, I've never used this plugin, and what I describe above doesn't seem to be an elegant solution. Your question specifically ask for a Jenkins plugin, however I believe the general problem is how to cache job dependencies in Jenkins. If you agree to this wider interpretation, there are some other options.

I have often solved the problem you mention using a combination of managed scripts and persistent storage which Jenkins and workers have access to, in Jenkins. If you need really arbitrary dependency management, a tool like Spack or EasyBuild may be useful. Although these have evolved in the HPC domain, they do quite well at caching dependencies for workloads.

However, since you mention just a few dependencies (specific sqlite versions e.g.), I think it may be worth your time to write a script which does the caching for you and, together with globally-available persistent storage, do the caching and retrieval that way.

1
  • That seems like a means to cache my own artifact -- file produced by my build. What I'm looking for is a way to efficiently download somebody else's tarball -- from afar... I know, I can write a script of my own, but I'm puzzled, why -- given their wide variety -- there is no plugin for this among the existing ones. FreeBSD's ports-collection's been doing just this (downloading, verifying, and building 3rd-party sources) for decades, for example...
    – Mikhail T.
    Commented Nov 5, 2019 at 5:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.