Best Practice for Container Image Versioning

Question

I'm looking for some guidance on best practices for container image versioning and despite reading a number of the best practice guides they leave some ambiguity.

I understand that most container images use Stable versioning and that the version tag will stay the same when things like frameworks, packages, libraries, etc are updated, but I'm unclear how this strategy ensures that the containers deployed from an image are consistent.

If I build an image today with the tag set to v1.0.0 and then next month the image is updated to include updates to frameworks, packages, libraries, etc but the application remains unchanged then my understanding of the stable versioning strategy means the new image will keep the tag set to v1.0.0.

With this strategy how do I ensure the same version of the image is deployed consistently? Developers, test environments, and production could all believe they are using the same version of the image as the tag is still set to v1.0.0 but changes in underlying frameworks, packages, and libraries could introduce instability which everyone is unaware of between environments. Presumably this would be even worse if there is a cluster running multiple container instances of a image.

One other dimension is that the frameworks, packages and libraries might be in a parent image and with no changes in the application image itself.

Whilst I could include a build number in the tag for the application images and any parent images that contain the frameworks, packages and libraries, this feels like a huge maintenance overhead as I'd need to create a new dockerfile for the application image every time the parent image is updated due to the parent image tag now including a build number e.g. from parent-image:v1.0.0-build123.

Is there a better strategy to solve this problem, or is it simply better to accept that there will be some differences between images using the same stable version tag? I'm not using Kubernetes at the moment but its something that we're likely to use in the future and I'm keen to pick a versioning strategy that will work for Kubernetes and ensure that there aren't "hidden" differences in the container instances deployed on the nodes from the same image version.

Answer 1

First a couple key points:

1. Tags are mutable in most registries, so v1 today may be different from v1 a month ago. However clients do need to pull this tag again to see the updates.
2. Multiple tags may point to the same image. So stable and v1 could point to the same image.
3. Pointing to a digest ensures the image is immutable, however you will not receive updates, and if that image becomes untagged, it's possible for some registries to delete it as part of their garbage collection.

With that said, the best practice depends on how you intend to use the image. For something that can be rebuilt in the future and receive security updates, you typically want to depend on a mutable but stable tag. This is particularly important when you don't have some way to automatically update your Dockerfiles with new image tags. However if you control the various ends of the build pipeline, you may want to depend on a digest for relatively reproducible builds and automatically inject that digest when your base image pipeline finishes.

In general, semver in docker tags is handled by building a persistent tag that the image creator intentionally doesn't change. So when 1.0.0 is released and tagged in git, that git tag is not changed, and the resulting image is v1.0.0 and it is also never changed (by policy). And with the ability to push multiple tags, that same build would also push v1.0 and v1. Then when 1.0.1 is tagged in git, the image v1.0.1 is built and the tags for v1.0 and v1 are modified to point to that new image.

That semver strategy allows for others to depend on the version they need in their Dockerfile with FROM base-image:v1 so that they will get minor updates, bug fixes, and security patches, without any modification to their Dockerfile, they just need to periodically rebuild and pull updated base images.

When upstream makes a breaking change, they release 2.0.0 in git, tag their image v2.0.0, v2.0, and v2. And anyone that depends on v1 will not be impacted by that breaking change until they update their Dockerfile to use the new major release.

Answer 2

The most stable way to assure you're deploying a consistent image is to use the digest instead of tags. The digest is like the git sha of a container image. This assures that if someone uploads a new image and tags it with an existing version tag, either accidentally or maliciously, you will continue to pull the known good image. The drawback is digests are not user-friendly.