A build-pipeline often requires to access third party resources, aside from the source code of the artefacts it is meant to build. Each of these third party resources introduce a risk factor in builds, for instance:
Distribution repositories are temporarily unavailable, therefore installing distribution packages temporarily fails.
Distribution repositories are permanently unavailable, same and worse as before.
Some NPM free-software package stops to exist, because its author reclaimed that withdrawal.
Some source file or binary ad-hoc package for a third-party tool is not available anymore.
This list could be extended ad lib and while studying the various reasons leading to some third-party resource to disappear is much more entertaining as it sounds, it first of all has for us a disastrous consequence: the build-pipeline is broken.
What the is the best way to cover the two following cases when building docker images:
- Caching files downloaded with a curl or wget.
- Caching Debian packages.
There are several plausible strategies that can be considered here. For instance using a proxy and configuring the build system to route all its requests through this proxy . A second, cheap, possibility would be to wrap curl calls to store results in a local cache.
(In this question I would consider sharing the cache among several units of the build pipeline as a question of secondary importance.)