Container Image Promotion across environments in ECR/ECS

Question

We have started doing some initial work on deploying containers on ECS Fargate service.

However we are not yet completely figured out how to manage our image promotion across different environments.

In our case, developers managed their own environment so they can deploy the docker image specifying just the tag of something they have built.

We only have UAT and PROD environments , As of now whatever we have in develop branch got baked in as image and pushed to ECR as develop tag and that gets deployed to UAT.

we release the code from develop to master as part of our release process.

During the release process ( from develop to master merge ) we create another container image this time on merge to master and that gets pushed to ECR as master tag which gets deployed to PROD.

I think this process is not ideal and it can be improved by a lot margin.

1. We shouldn't need to create image on each stage and should promote the same image tag in across environments ( i.e whatever we have in UAT will get QA/TESTED and released to PROD as artifact ), how are you managing this image promotion across environment ? what are other release process look like ?

2. We are currently tagging our image as branch tag i.e develop or master , How are you managing these tags creation ? Are you using GIT SHA ? or Sem ver ?

Any other improvements or general comments can be really helpful as well.

Answer 1

1. We shouldn't need to create image on each stage and should promote the same image tag in across environments ( i.e whatever we have in UAT will get QA/TESTED and released to PROD as artifact ), how are you managing this image promotion across environment ? what are other release process look like ?

Yes, rebuilding images is considered an anti-pattern in containerized world. While Bruce's solution with re-tagging in a sibling answer is neat, it doesn't solve rollbacks.

Recommended approach for production is to pin your images to their sha256 digests. So image definition in task would be something like myregistry/myimage@sha256:shadigestgoeshere. This ensures maximum degree of control what image gets deployed.

In some cases people update those tags manually, which is pretty tedious and brittle. However, there exists specialized tooling which keeps track of versions and routes what images should go there (dislaimer: I'm working on a tool of this sort - Reliza Hub).

2. We are currently tagging our image as branch tag i.e develop or master , How are you managing these tags creation ? Are you using GIT SHA ? or Sem ver ?

It is common to use version, git sha and branch as image tags. It is fine to have same image tagged with all of them, so in example you may have all of the following for same image:

• myregistry/myimage:mybranch
• myregistry/myimage:0.1.0
• myregistry/myimage:mygitsha

This can be achieved very easily by just adding multiple -t flags on docker build command. Again, remember that images under same tag can mutate (unless you use some special constraints on docker registry side which is rare) - so the only good way to ensure exact image getting deployed is using sha256 digests in production manifests.

Any other improvements or general comments can be really helpful as well.

It is recommended to label images with version, branch, git commit, some ci metadata. This way when you do image inspect, you know right away what it is. See my toy project here for an example how to do labelling via build args - https://github.com/taleodor/mafia-vue .

Answer 2

Although difficult to give a canonical answer, I think I understand your frustration. My opinion is that you are focussing on promoting the wrong thing - the image.

I would start with the concept of "Dev-Prod parity" (or in your case, UAT/Prod parity). This concept says:

Keep development, staging, and production as similar as possible

The simplest way to do this is to use the same image in the two environments. The difference between them comes from the configuration and backing services. These are set at run time, not at build time - ie they are changed when you start the container not when you build it.

So, your process should look like:

1. Build the image, tag it with uat
2. UAT environment discovers new image, new image gets deployed
3. Run UAT against new deploy, they pass
4. tag that same image with latest
5. Prod discovers new image, gets deployed.

This process is extremely lightweight and you will have certainty in what is deployed.

TL;DR instead of promoting images and having to rebuild them, just shift the tag used in prod.