How can I deal with container "patching" in a strict release process?

Question

I am building a new Container and Kubernetes system for my company. Overall it is going well, but container patching has me a bit confused.

We follow a fairly strict release process. New versions are tested in Dev, then promoted to Test (where they get integration tested). While we sometimes have some other environments, the basic flow then promotes to production. All of this is done using the binaries that are built in Dev (no rebuilding once it leaves Dev).

The version of the application then runs in production until that application needs another version (we have a lot of applications).

Currently (with our VM structure) we will perform patching on the VMs that host the application. While some testing is done for more critical apps, mostly the theory is that the application code is not changed, so it will be OK as security patches are applied to the VM. We occasionally have issues doing this, but it is very very rare.

Now enter container images. They are immutable. You don't apply security patches container images. You build new ones.

This causes problems because, the container image has the base image and the application in one "binary". To update the base image, you have to rebuild the whole thing. That means a new binary, which means more testing is needed. (Even if I get the exact same version of the source, it is not the same binary.)

I see this potentially exploding our deployment and testing pipeline.

Are there strategies for dealing with this?

Answer 1

I have tackled a very similar problem. We also do not want to rebuild when we patch. To solve this problem, when we do a build, we store our artifact in a repository such as jfrog or nexus. Then when it comes time for patching, we fetch the artifact from the repository and put it on top of an updated base image.

How we use base images is also a very important factor. We rely on stable base image tags such as openjdk11:latest and trust the vendor that any updates are safe since it is still jdk11. This of course means selecting a reliable base image. There are several options here and one worth checking is RedHat UBI images.