Helm is a Kubernetes tool so this answer assumes you are deploying to Kubernetes or OpenShift not Docker Swarm.
If you are using Helm well then I don't think it is particularly important what build pipeline tech you are using to invoke it. I think what is more important is how you manage all your configuration within Git and how you structure the Helm deployments.
There are two different paradigms you can follow:
- Build the app container image with the correct environment variables baked within it for each environment.
- Build only one app container image that picks up environment variables from within the environment it has been deployed.
If you follow 12factor.net best practices then you go for Option 2. IMHO it feels far cleaner and less error-prone to move the exact same image between kubernetes namespaces that represent different environments. We store environment variables inside of Kubernetes as secrets or configmaps. We then get to two more questions:
- How do you load
secrets and configmaps as environment variables into your .Net Core runtime images when the container starts up?
- How to you manage the
secrets and configmaps in each environment?
With respect to the first question, Kubernetes has this covered. Any deployment can name a bunch of different secrets that should be loaded as environment variables. Ditto for configmaps. You just need each environment to be a different namespace so that the same logical secret can have the same name in each environment. Your database credentials can then be a secret called main-database-credentials that holds different credentials in different environments.
To deal with the second question we use Helm to manage both secrets and configmaps. We have one generic Helm chart that deploys all our different secrets in all our different environment:
helm upgrade --install -f $NAME.yaml $NAME secret-1.1.tar.gz
So to upgrade our main-database-credentials secret we would change the settings in main-database-credentials.yaml then tell the chart to recreate the main-database-credentials deployment using that command. The chart simply uses the Helm templating language to build yaml that defines the secret using the name set in the yaml file. Likewise, we have a generic chart that updates configmaps that is invoked with a very simlar command as above. Helm uses a standard Golang templating library so it is very straight-forward to create those generic charts.
At first, it may seem a bit heavyweight to use Helm to update something like a database password secret. Yet helm gives you a deployment history and lets you diff the history of deployments. It also lets you easily roll-back to a previous deployment. That is really powerful as an audit trail of config changes that is really helpful when managing a complex microservices deployment. Since we also use Helm to deploy all our apps it makes perfect sense to use the same power tool to manage the deployment of configuration into our environments.
As we can use a generic helm chart to deploy many secrets into many environments why not use a generic helm chart to deploy all our microservices? We do:
helm upgrade --install -f $NAME.yaml $NAME deployer-1.3.tar.gz
This means that NAME=my-microservice and NAME=my-admin-webapp or anything else can be deployed using one generic deployer chart. The path and tag of the application container image to deploy is in the yaml file. The container path is different for different apps and the container tag can be different between environments.
That might sound strange as normally Helm packages a particular app so you would expect a one-to-one relationship between charts and apps. Yet if all your microservices are containers following 12factor.net best practices shouldn't they all "look the same" from the perspective of deploying them; it is just the container and the environment variables you deploy into the one generic chart. I see it that the app is the pure container with none of the configurations to run it and the Kubernetes configuration to run it is "application infrastructure". So I have a Helm chart for each type of application infrastructure. I don't want each bit of application infrastructure to be hand-rolled I want to cookie cutter create it using Helm.
Now we have covered all the Helm mechanics we are left with the question "how do we manage all our configuration within git?". We use different repo for different purposes:
- Each microservice has its own repo. We use CircleCI to build the code into an image that is configured via environment variables. We don't store any Helm or Kubernetes config in this repo. We do have a properties file that names the latest version of the
deployer chart this code is compatible with.
- We have a GitHub repo that holds the source of the generic Helm charts that are used to deploy all our apps and secrets. These don't actually change very often now they are fairly stable. CircleCI builds and releases the charts as tar.gz onto the GitHub releases page for this repo.
- We have a git repo per environment. This has a folder per deployed microservice within that environment. Each contains the
yaml file to pass to the helm deploy encrypted with git secret. It has a folder per deployed secret within that environment. Each contains a yaml file that is git secret encrypted. Ditto for configmaps.
Why a repo per environment? A branch per environment sounds more normal. Only a database credential for the live environment never moves between environments. As we use a separate repo for live we can give it separate permissions. We might let all developers go wild in the dev-environment repo we have a bit more discipline with the live-environment repo.
We then have CircleCI do automated deployments of configuration. This works as Helm data detects changes and doesn't deploy a change to any Kubrernetes object if the latest yaml it generates is already deployed. To change a password in live you change it in the git secret encrypted yaml in the live repo and apply a release tag. CircleCI pattern matches on the tag name to see that its a release tag and it runs the deploy step. That uses a container with the helm client to run a bash script that loops over all app folders, secret folders and configmap folders running Helm for the right chart and correct yaml. Helm will only apply updates where there are changes within git.
