My teams develop on their laptops (using the wonderful K3D) with a local repo. But, the test and prod environments are in Azure. Locally we configure a docker image repository (using k3d), and in Azure we use ACR. My teams' internet connections are flakey at best, so we cannot all be online all the time.
Currently, I use an environment variable to manage this. In my k8s config files I use the env-var name template pattern:
In my makefile I have this:
@cat k8s.yaml | envsubst | sed "$$CERT" | kubectl apply -f -
Which substitutes the current variable value into the k8s file before applying it to k8s. I use the same pattern in my azure clusters (with Tekton) where I configure the tekton pod with env-vars that are populated from a config map.
This means that I can use the local registry when working locally and I can use ACR when running in azure, and the only change is in the environment.
I think this is quite 'linuxy' and it works, though it does have its flaws (it's quite easy to make a frustrating but non-fatal mistake by not sourcing the correct environment before one begins development). And, the registry isn't the only difference (it's just the one I'm using here); we also have differences with certs (prod is all tls, dev is no tls), and ingress-class (nginx
addon-http-application-routing in azure), which has led to some more complex processing before applying my configs in places.
I'm worried that this is becoming too complicated. It's just about manageable right now, but if I add a couple more environment-based differences it'll start getting out of hand. Does anyone have a better suggestion that isn't "Use Helm"? I'm not opposed to helm at all, it's just that I already know that I should be using helm but it seems like such a heavyweight solution to simply swapping a couple of strings and deleting a line or two.