I am building a new Container and Kubernetes system for my company. Overall it is going well, but container patching has me a bit confused.
We follow a fairly strict release process. New versions are tested in Dev, then promoted to Test (where they get integration tested). While we sometimes have some other environments, the basic flow then promotes to production. All of this is done using the binaries that are built in Dev (no rebuilding once it leaves Dev).
The version of the application then runs in production until that application needs another version (we have a lot of applications).
Currently (with our VM structure) we will perform patching on the VMs that host the application. While some testing is done for more critical apps, mostly the theory is that the application code is not changed, so it will be OK as security patches are applied to the VM. We occasionally have issues doing this, but it is very very rare.
Now enter container images. They are immutable. You don't apply security patches container images. You build new ones.
This causes problems because, the container image has the base image and the application in one "binary". To update the base image, you have to rebuild the whole thing. That means a new binary, which means more testing is needed. (Even if I get the exact same version of the source, it is not the same binary.)
I see this potentially exploding our deployment and testing pipeline.
Are there strategies for dealing with this?