How do I run a CI build in a docker image matching the current `Dockerfile` while being resource-aware?

Question

Given a repository containing a Dockerfile that defines the build environment used by the CI pipeline as well as by the developer (e.g. as a Visual Studio Code devcontainer), the CI pipeline shall fulfill the following requirements:

1. The CI pipeline shall build the repository's contents in the context of a docker image built from the very Dockerfile present in the working branch (or the result of its merge to the target branch where applicable in the context of a pull request).

• Rationale: Builds shall be deterministic so that old repository versions shall be built with their original docker image, not with the newest one.
• Rationale: Changes to the Dockerfile shall be considered automatically without the need for manual user intervention (i.e. local docker build followed by a manual push).

2. The CI pipeline shall reuse an already built docker image if the applicable Dockerfile has already been "baked" into such an image pushed to the registry.

• Rationale: Building a docker image on each commit is resource-consuming (time, energy).

I couldn't find an existing best practice to implement that out of the box (my CI environment being Azure DevOps Pipelines if it matters) so I came with the following concept:

• Calculate the Dockerfile's hash.
• Load docker_image_name:$hash from the docker registry.
• If the load fails, build docker_image_name:$hash from the Dockerfile and push it to the docker registry.
• Use docker_image_name:$hash (from the registry / from the local cache) to run the CI pipeline (using Azure's container jobs in my case).

Questions:

1. Does this procedure make sense as a solution to my use case?
2. I can't imagine being the first to realize this use case. Is there an existing mechanism (as part of the docker utilities, as part of the Azure DevOps Pipelines framework or from something completely different) that fulfills my needs?

Answer 1

Does this procedure make sense as a solution to my use case?

vscode devcontainers are one of the greatest inventions in the DevOps space and I can't believe more people are not talking about them. I think having the local dev environment and the pipeline in sync makes a lot of sense. Though I think the procedure you outlined above could be simplified.

I can't imagine being the first to realize this use case. Is there an existing mechanism (as part of the docker utilities, as part of the Azure DevOps Pipelines framework or from something completely different) that fulfills my needs?

I'm currently doing this in some of my side projects. It is also supported on azure.

Answer 2

Docker has this capability built-in and you can use it in a variety of ways.

One way would be to keep a "cache" tag for every build. This is good if your repo normally doesn't have a lot of parallel work being done.

docker pull ${MY_IMAGE}:cache || true
docker build --cache-from ${MY_IMAGE}:cache -t ${MY_IMAGE}:${VERSION} -t ${MY_IMAGE}:cache .
docker push ${MY_IMAGE}:${VERSION}
docker push ${MY_IMAGE}:cache

If your repo does have a lot of parallel work being done, don't create a cache tag, instead use the latest version.

docker pull ${MY_IMAGE}:latest || true
docker build --cache-from ${MY_IMAGE}:latest -t ${MY_IMAGE}:${VERSION} .
docker push ${MY_IMAGE}:${VERSION}

This is basically the following the process you outlined in your question but instead, we are relying on docker to do the work for us. The first step is to pull the image layers down locally. This is faster (usually) than building a new image and if your CI/CD is setup correctly should already be present for the use of caching.

Then the --cache-from flag tells Docker to use that image as cache. Which will prevent Docker from building any layers that haven't changed.