I want to run CI processes that depend on a vendor application which is released periodically as a new ZIP file (about 500MB), downloaded from behind a login on the vendor's website. When a new version of this application is released, I want to be able to download the ZIP to my local machine and then upload it to a private asset store, kicking off CI processes that do things like running integration tests for my custom modules or building privately-hosted docker images for deployment to staging environments.
I am struggling to figure out the most effective way of storing these ZIP files such that they can be consumed by scripts in repositories hooked up to CI infrastructure. I would like to follow twelve-factor principles as much as possible. Here is my thinking on this issue so far:
What I think I want
Conceptually, I think the ideal solution would be to use some kind of language-agnostic artifact registry that supports private access. In this model, I would have a simple manual process to go through every time a new version of the application is released, whereby I would download the new ZIP file from the website and upload it to the artifact registry under the new version number.
That would theoretically be the only guaranteed-manual process: automatic CI tools like Snyk or Dependabot would then notice that the latest artifact version has changed, and begin running jobs on my repositories against the new artifact version. On the receiving end, my build scripts (either in a CI environment or running in my dev environment) would invoke a client for this artifact registry which downloads the artifact into a subfolder of the working directory (this subfolder would be
.gitignored for dev and cached in the CI environments), making it available for consumption by those build scripts.
I am currently unaware that such a language-agnostic artifact registry exists. I initially had high hopes for Google Cloud Platform's Artifact Registry, but on further inspection this only provides hosted versions of existing registries like Docker, NPM, and Maven. (In other words, it is not a new language-agnostic artifact registry platform.)
What I think my (sub-optimal) options are
I can think of two possible options based on technology I'm already familiar with. One would be to repurpose some existing registry/package manager technology to handle this opaque ZIP file asset, and just not use any of the package management functionality that integrates module code with a project. Part of me wants to use Docker for this, by uploading a from-scratch docker image that contains only the ZIP file to a private registry. On the receiving end, most CI tools allow the use of docker images as the entire build environment, or as a service dependency (like a database or caching layer). In the former case, I would need to build in all of the rest of my tooling into the image, which defeats the purpose of having this artifact as an uncoupled dependency. In the latter case, I don't know of any docker functionality that exports a container's contents as a filesystem for use by other containers (this also feels rather counter to containerization philosophy), but I could conceivably build a docker image that provides a web server endpoint that can allow other processes running on the same host to "download" these ZIP files from it quickly. This feels... clunky, however.
The other option would be to DIY a rudimentary private "registry" using some password-protected S3-style object storage and HTTP calls from cURL or Invoke-WebRequest (or perhaps an S3 API client). I think I probably could do this successfully, but... this can't be the optimal solution, can it?
I've also recently started thinking about Git-LFS as a possible solution (I am not very experienced with Git-LFS but I understand the concept), but I would not want to include the ZIP file itself in each repo that consumes it, since this requires more manual updating every time a new version comes out. I could see potentially having a single repo contain the ZIP files using Git-LFS, and then including this single repo as a submodule of all of the projects that depend on it. I am not afraid of git submodules, but I do not know how they interact with Git-LFS, and I don't know what kind of support tools like Snyk/Dependabot have for updating branch-tracked submodules.
What I've tried
Previously, my solution for keeping these ZIP artifacts separate from my code has been to store them in a developer-accessible file share (on LAN network-attached storage), and pass the path of this share to my build tool using an environment variable. But this file share isn't particularly accessible to CI tools running in the cloud. I could see replacing this with Google Drive or a similar service, but them I'm still in the position of basically implementing a package manager client to operate as part of my build scripts. (And then probably packaging that client up and distributing it from a package registry...)
While writing this, I've also come across some other tools like JFrog Artifactory and Sonatype Nexus, which may be helping me aim in the right direction ("Binary artifact repositories").