How to cache third party resources in a build pipeline?

Question

A build-pipeline often requires to access third party resources, aside from the source code of the artefacts it is meant to build. Each of these third party resources introduce a risk factor in builds, for instance:

• Distribution repositories are temporarily unavailable, therefore installing distribution packages temporarily fails.

• Distribution repositories are permanently unavailable, same and worse as before.

• Some NPM free-software package stops to exist, because its author reclaimed that withdrawal.

• Some source file or binary ad-hoc package for a third-party tool is not available anymore.

This list could be extended ad lib and while studying the various reasons leading to some third-party resource to disappear is much more entertaining as it sounds, it first of all has for us a disastrous consequence: the build-pipeline is broken.

What the is the best way to cover the two following cases when building docker images:

1. Caching files downloaded with a curl or wget.
2. Caching Debian packages.

There are several plausible strategies that can be considered here. For instance using a proxy and configuring the build system to route all its requests through this proxy . A second, cheap, possibility would be to wrap curl calls to store results in a local cache.

(In this question I would consider sharing the cache among several units of the build pipeline as a question of secondary importance.)

Answer 1

I'm not overly familiar with Docker particularly but it sounds a lot like issues one would get when building a normal .NET application or java app with the need to store their external dependencies (DLL or jar).

To solve this I'm quite fond of artifactories. Two of the most popular are sonatype's Nexus and JFrog's Artifactory. Both offer a free Open Source version and both will have some way to store customised package or use it as a raw depot. There are other options out there, I just listed these two because I've worked with them already and know they will do what you need. As a bonus you might also be able to use them to manage the docker image directly as well.

There are others too worth mentioning, you can find a comparison table for them, you'll get a better view of their capabilities. (thanks to Karl Harnagy for the link)

Note that some of these options might require one or another of the paid versions so YMMV.

Some continuous integration systems will also support the concept of artifacts (from experience Jenkins and team-city do this, others might as well though) which should allow you do achieve similar results directly.

If none of these do it for you you can look into other technologies or roll your own. I personally like to leverage whatever I already have this reduces the maintenance required in long term and makes passing the buck easier when comes time to look for new challenges elsewhere.

Hope this helps

Answer 2

It looks like what you're looking for is less about caching and more about mirroring. Because your requirements have to deal with permanently unavailable packages, you need to persist the packages yourself in something more permanent than a cache. In the past, we've stored packages in persistent, cloud-based stores like S3. However, there is nothing to preclude you from setting up a custom filestore on the build server.

More concretely, it's probably easiest to set up a proxy service that looks for a resource in your filestore (S3) and returns the package if it's found. Otherwise, it would just request the resource from the upstream, populate your filestore, and return the package.