1

My current client uses .net and .net core, tfs and deploys via octopus. They want to use helm, k8s and docker. With all of the environments, env variables are already in octopus.

I am thinking to restore the nuget packages in the dockerfile, build the image, push to our local registry. Add our docker registry as a feed, add helm as a feed then creating a lifecycle for dev and prod. Then modify the octopus jobs to point to the local registry to pull the images.

Anyone see any issues here? Should I strip out the env variables from octopus and just get rid of octopus?

|improve this question
0

Helm is a Kubernetes tool so this answer assumes you are deploying to Kubernetes or OpenShift not Docker Swarm.

If you are using Helm well then I don't think it is particularly important what build pipeline tech you are using to invoke it. I think what is more important is how you manage all your configuration within Git and how you structure the Helm deployments.

There are two different paradigms you can follow:

  1. Build the app container image with the correct environment variables baked within it for each environment.
  2. Build only one app container image that picks up environment variables from within the environment it has been deployed.

If you follow 12factor.net best practices then you go for Option 2. IMHO it feels far cleaner and less error-prone to move the exact same image between kubernetes namespaces that represent different environments. We store environment variables inside of Kubernetes as secrets or configmaps. We then get to two more questions:

  1. How do you load secrets and configmaps as environment variables into your .Net Core runtime images when the container starts up?
  2. How to you manage the secrets and configmaps in each environment?

With respect to the first question, Kubernetes has this covered. Any deployment can name a bunch of different secrets that should be loaded as environment variables. Ditto for configmaps. You just need each environment to be a different namespace so that the same logical secret can have the same name in each environment. Your database credentials can then be a secret called main-database-credentials that holds different credentials in different environments.

To deal with the second question we use Helm to manage both secrets and configmaps. We have one generic Helm chart that deploys all our different secrets in all our different environment:

helm upgrade --install -f $NAME.yaml $NAME secret-1.1.tar.gz

So to upgrade our main-database-credentials secret we would change the settings in main-database-credentials.yaml then tell the chart to recreate the main-database-credentials deployment using that command. The chart simply uses the Helm templating language to build yaml that defines the secret using the name set in the yaml file. Likewise, we have a generic chart that updates configmaps that is invoked with a very simlar command as above. Helm uses a standard Golang templating library so it is very straight-forward to create those generic charts.

At first, it may seem a bit heavyweight to use Helm to update something like a database password secret. Yet helm gives you a deployment history and lets you diff the history of deployments. It also lets you easily roll-back to a previous deployment. That is really powerful as an audit trail of config changes that is really helpful when managing a complex microservices deployment. Since we also use Helm to deploy all our apps it makes perfect sense to use the same power tool to manage the deployment of configuration into our environments.

As we can use a generic helm chart to deploy many secrets into many environments why not use a generic helm chart to deploy all our microservices? We do:

helm upgrade --install -f $NAME.yaml $NAME deployer-1.3.tar.gz

This means that NAME=my-microservice and NAME=my-admin-webapp or anything else can be deployed using one generic deployer chart. The path and tag of the application container image to deploy is in the yaml file. The container path is different for different apps and the container tag can be different between environments.

That might sound strange as normally Helm packages a particular app so you would expect a one-to-one relationship between charts and apps. Yet if all your microservices are containers following 12factor.net best practices shouldn't they all "look the same" from the perspective of deploying them; it is just the container and the environment variables you deploy into the one generic chart. I see it that the app is the pure container with none of the configurations to run it and the Kubernetes configuration to run it is "application infrastructure". So I have a Helm chart for each type of application infrastructure. I don't want each bit of application infrastructure to be hand-rolled I want to cookie cutter create it using Helm.

Now we have covered all the Helm mechanics we are left with the question "how do we manage all our configuration within git?". We use different repo for different purposes:

  1. Each microservice has its own repo. We use CircleCI to build the code into an image that is configured via environment variables. We don't store any Helm or Kubernetes config in this repo. We do have a properties file that names the latest version of the deployer chart this code is compatible with.
  2. We have a GitHub repo that holds the source of the generic Helm charts that are used to deploy all our apps and secrets. These don't actually change very often now they are fairly stable. CircleCI builds and releases the charts as tar.gz onto the GitHub releases page for this repo.
  3. We have a git repo per environment. This has a folder per deployed microservice within that environment. Each contains the yaml file to pass to the helm deploy encrypted with git secret. It has a folder per deployed secret within that environment. Each contains a yaml file that is git secret encrypted. Ditto for configmaps.

Why a repo per environment? A branch per environment sounds more normal. Only a database credential for the live environment never moves between environments. As we use a separate repo for live we can give it separate permissions. We might let all developers go wild in the dev-environment repo we have a bit more discipline with the live-environment repo.

We then have CircleCI do automated deployments of configuration. This works as Helm data detects changes and doesn't deploy a change to any Kubrernetes object if the latest yaml it generates is already deployed. To change a password in live you change it in the git secret encrypted yaml in the live repo and apply a release tag. CircleCI pattern matches on the tag name to see that its a release tag and it runs the deploy step. That uses a container with the helm client to run a bash script that loops over all app folders, secret folders and configmap folders running Helm for the right chart and correct yaml. Helm will only apply updates where there are changes within git.

|improve this answer
  • 1
    Thank you. Great info here. The k8s and docker env was built before i started and they arent using it yet. My real question is do i replace octopus or take advantange of whatvis already There and use tge new festures that allow you to deploy to k8s. This customer really has no vlue what they wsnt. But the seem to believe they shoukd replace octopus alltogether with k8s and docker. I think we use what we have and ad k8s and docker to octopus – Just a guy Nov 10 '18 at 20:21
  • old microsoft tech wasn't 12factor friendly. octopus was liked by teams i knew as solved repackaging same code "built once" for each env. that isn't a problem with .net core and k8s. you only got a few tentacles for free their so they liked octopus but couldn't use it on microservices as not enough free capacity. yet if it is working for your team today why change it immediately? how config is managed and whether the same container moves through envs that devs can run locally overriding vars is most important part. get it working that way and remove octopus last if its not helping. – simbo1905 Nov 11 '18 at 7:37
  • for "build once, deploy many times" I strongly recommend Helm. it doesn't really matter what is invoking Helm. we use a CircleCI deploy step, in the past, I have had VSTS deploying to k8s with bash script. developers have strong opinions the about "build" solution and SCM but are not bothered about deploy tooling if it "just works". CircleCI works for us as it integrates with GitHub and lets me run a helm container in the deploy step with a simple bash script. – simbo1905 Nov 11 '18 at 8:08
  • 1
    Thank you. Its been very difficult finding info. Mostly because people usually choise one or another.as of now i am creating a new octopus server to test the octopus to k8s deployment since this team knows tfs and octopus. They no nothing about hhelm k8s or docker. Once i hsve a test env. I will do 1 in octopus and one without octopus – Just a guy Nov 11 '18 at 16:26
  • Im standing up a new env in parallel to what thry currently use. So i can build it however i want. So tfs to dockerfile, build .net in container" image push to local registry. Deploy to k8s via helm – Just a guy Nov 11 '18 at 16:57

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.