Local dev, online test/prod - best approach?

Question

My teams develop on their laptops (using the wonderful K3D) with a local repo. But, the test and prod environments are in Azure. Locally we configure a docker image repository (using k3d), and in Azure we use ACR. My teams' internet connections are flakey at best, so we cannot all be online all the time.

Currently, I use an environment variable to manage this. In my k8s config files I use the env-var name template pattern:

image: $K8S_REPO_URL/dev/tenants

In my makefile I have this:

@cat k8s.yaml | envsubst | sed "$$CERT" | kubectl apply -f -

Which substitutes the current variable value into the k8s file before applying it to k8s. I use the same pattern in my azure clusters (with Tekton) where I configure the tekton pod with env-vars that are populated from a config map.

This means that I can use the local registry when working locally and I can use ACR when running in azure, and the only change is in the environment.

I think this is quite 'linuxy' and it works, though it does have its flaws (it's quite easy to make a frustrating but non-fatal mistake by not sourcing the correct environment before one begins development). And, the registry isn't the only difference (it's just the one I'm using here); we also have differences with certs (prod is all tls, dev is no tls), and ingress-class (nginx locally, addon-http-application-routing in azure), which has led to some more complex processing before applying my configs in places.

I'm worried that this is becoming too complicated. It's just about manageable right now, but if I add a couple more environment-based differences it'll start getting out of hand. Does anyone have a better suggestion that isn't "Use Helm"? I'm not opposed to helm at all, it's just that I already know that I should be using helm but it seems like such a heavyweight solution to simply swapping a couple of strings and deleting a line or two.

Answer 1

My suggestion is "use kustomize", and then create a development overlay that overrides the images. Using the image transformer, you would end up with something like:

# This assumes that your deployment manifests are all locaed in the
# directory ../production. What we're doing here is including all
# of those manifests and then applying some transformations.
resources:
  - ../production/

images:
  - name: appimage
    newName: local_repo_url/appimage

This would replace all references to an image named appimage (in Pods, Deployments, Statefulsets, Daemonsets, Jobs, etc) with local_repo_url/appimage. You can also transform the tag, etc.

I really like working with kustomize (I use it both for work and personal projects), but it does require to be disciplined with respect to how you lay out your manifests.