How to manage 10+ team environments with Helm and Kubernetes?

Question

I'm looking for feedback on how we can improve our team environment and staging deployment process. We currently have 10 agile teams, and set up our Kubernetes deployments through Helm as follows:

All our helm charts are in a single git repo. At the top layer of the repo we have a collection of bash scripts (deploy.sh, buildSecrets.sh, and buildConfigMaps.sh) and some json configuration files. (namespaceCharts.json, secrets.json, configmaps.json)

The two main configuration files are a long list of secrets, which are encrypted, and a long list of config map environment variables, 80% of which are shared between helm charts. The namespaceCharts.json defines which charts need to be deployed into each namespace grouping.

For each environment that we deploy into, we clone the repo, change the values of the secrets.json and configmaps.json and then run the scripts passing in our base namespace name. (For example, the team name)

This results in us basically creating a series of namespaces such as teamA-frontend and TeamA-backend each of which contains the right microservices with the connection strings to the correct DB etc.

We are currently hitting a few main roadblocks.
1. It's hard to deploy to multiple clusters or new clusters.
2. It's hard to get a single source of truth for the configurations.
3. It's hard to set up a brand new Team environment because we have to go through the entire configuration file to see what needs to be changed or kept.

I have come up with three ways of moving forward, all of which "feel wrong" to me.
1. Store all the environment configurations in separate "configurations" git repo.
2. Create a configuration service with a DB table defining each environment, and create a pub/sub feature to dynamically grab configuration at runtime, as well as dynamically grab configurations in the .sh scripts.
3. Use ansible or something similar to store the scripts and configurations for each environment.

Are we doing this all wrong? Are we missing some obvious best practice to do this better? Are one of those three ideas actually the right approach?

Answer 1

My 2 cents. We have similar requirement like you do. We end up having multiple values files in the form of values-<namespace>.yaml and it should come with the defaults set to values.yaml

We also have some shell scripts to deploy to the namespace. Something like this deploy.sh --namespace app.

It is not without its problem, the number of files are steadily growing. But so far we are ok with it.

Answer 2

Checkout flux(https://github.com/weaveworks/flux/), it introduces a new kubernetes crd called HelmRelease, in which you can include your helm release info(like the chart repo, name, version, namespace, and values. You can even pull your charts from a git repo).

The flux controller monitors a git repo of your choice(you can monitor a path inside the repo as well, which I think might best suit your requirements). And applies all the yaml/json files(including HelmRelease definitions) found in that path periodically.

When a new HelmRelease definition is created, this other component of flux called helm-operator takes over and try to create/update your helm release to the desired state.

The nice thing about this is you can define the desired cluster state using various yaml/json files and flux will try to keep it in sync. If you want to crate a new cluster, just copy the files to a new folder, deploy your cluster, configure flux on it with the new folder path and it'll do the rest of the work for you. Mind you flux is not a full ci/cd solution, so you can't run your shell scripts in it, you will have to find a different way to encode that information into yaml files.

Take a look here(https://github.com/stefanprodan/gitops-helm) for a minimal git repo.

Answer 3

I have made the scripts and tools that we use to solve these problems opensource on GitHub calling it OCD. You mention that you have thought the following feels wrong:

• Store all the environment configurations in separate "configurations" git repo

• Use ansible or something similar to store the scripts and configurations for each environment.

With OCD we use Helmfile which makes it easy to manage many helm release in either one of a small number of simple yaml file. If you are familiar with Ansible then it gives all the benefits of that tool but it only manages helm releases. It makes total sense to put the helmfile.yml configuration into Git. These act as the playbooks to setup your environment. They shouldn't live in the application repos it should be in one or more "environments repo(s)".

We have created a generic chart to install secrets and another to install configmaps. Your folder structure might look like:

|____config
| |____nexmo.yaml.secret // <- gpg encrypted secret
| |____reepay.yaml.secret
| |____backoffice-db.yaml.secret
| |____helmfile.yaml // <- hemlfile manages 3 secret releases and inline configmaps
|____backoffice-app
| |____helmfile.yaml // <- hemlfile manages 1 app
|____frontoffice-app
| |____helmfile.yaml // <- hemlfile manages 1 app

In the above example, we have three helmfiles. There is one that manages shared configuration and one each for two apps that use the shared config. Since shared config is going to be multipe secret and configmaps it just uses the two generic charts. It uses the generic secret chart to stamp out many separately encrypted secrets using different values files. Likewise, one generic configmap chart is used to stamp out many configmaps by feeding it different values.

With OCD you setup webhooks on your config repo to point at an adnanh/webhook. When you push a change to the config repo the webhook fires. The webhook runs a script to checkout the config repo, decrypt the secrets, then does helmfile apply in each folder that has a helmfile.yaml. That does a helm diff on every release in each helmfile and so will upgrade any secrets, configmaps or application deployments where the yaml has changed.

We have a wizard script to install the webhook into a new environment. This means that a team can spin up a new environment using the config in git quickly. Since it is loading the config from git it is an accurate process. Here is a video of setting up a new environment in six minutes which would be quicker if I was not making a video.

Helmfile reduces the burden of having to go through all the files to try to find all the settings to change for a new team environment. The helmfile.yaml captures whatever you are overriding over the chart defaults. All the configmaps and secrets are organised to be easily found in one place. Helmfile has quite a lot of capabilities that make managing config and managing many helm releases easier.

We happen to be using openshift kubernetes so our application deployment chart for microservices uses openshift specific yaml. Yet the secrets and configmaps charts are vanilla k8s. So the OCD approach and scripts to manage config should work with any k8s distro. We plan to have a 1.0 release of OCD in the next couple of weeks. I was considering doing a demo of running OCD on vanilla k8s to manage configuration. So I would be really interested in your thoughts and perhaps we can collaborate. Just raise an issue on the ocd-meta github asking about running it on the version of k8s you are using.

Answer 4

I would recommend using something like ArgoCD - a declarative, continuous delivery for Kubernetes manifests.

With ArgoCD you can:

• structure the Kubernetes objects into separate repositories instead of one huge repo and therefore assign permissions as needed,
• set up RBAC permissions around ArgoCD projects, so each team can control only "their" objects,
• orchestrate deployments into multiple clusters from a central place if desired.

ArgoCD works with plain Kubernetes manifests, Helm charts, Kustomize dynamic manifests ... just to scratch the surface.