I'm looking for feedback on how we can improve our team environment and staging deployment process. We currently have 10 agile teams, and set up our Kubernetes deployments through Helm as follows:
All our helm charts are in a single git repo. At the top layer of the repo we have a collection of bash scripts (
buildConfigMaps.sh) and some json configuration files. (
The two main configuration files are a long list of secrets, which are encrypted, and a long list of config map environment variables, 80% of which are shared between helm charts. The
namespaceCharts.json defines which charts need to be deployed into each namespace grouping.
For each environment that we deploy into, we clone the repo, change the values of the
configmaps.json and then run the scripts passing in our base namespace name. (For example, the team name)
This results in us basically creating a series of namespaces such as
TeamA-backend each of which contains the right microservices with the connection strings to the correct DB etc.
We are currently hitting a few main roadblocks.
1. It's hard to deploy to multiple clusters or new clusters.
2. It's hard to get a single source of truth for the configurations.
3. It's hard to set up a brand new Team environment because we have to go through the entire configuration file to see what needs to be changed or kept.
I have come up with three ways of moving forward, all of which "feel wrong" to me.
1. Store all the environment configurations in separate "configurations" git repo.
2. Create a configuration service with a DB table defining each environment, and create a pub/sub feature to dynamically grab configuration at runtime, as well as dynamically grab configurations in the .sh scripts.
3. Use ansible or something similar to store the scripts and configurations for each environment.
Are we doing this all wrong? Are we missing some obvious best practice to do this better? Are one of those three ideas actually the right approach?