I've had two runs at doing environment variables in a scalable way and neither has ended up perfect because,  as I've discovered, is a very tricky thing to get right. I'll give a summary of both of my experiences below:

**Common Factors**

 - Environment variables are stored in a separate repository from the original source code (they are submoduled together but are still based on separate repos)
 - There is a serperate "build" process for the artifact and it's variables.
 - There is **not** a seperate release process for the environment variables. If you want to change environment variables you need to go through the same change review boards and usual

**Using Consul KV Pairs** 

The environment variables are loaded from an artifact repository (never the original git repo) and loaded into a namespaced KV pair tree, for example

/env/dev1/my/application/v1.1.1

Where the preceding dev1 is the name of the environment, the my/application is the application namespace, and the v1.1.1 is the version of the environment variables to use.

To developers all these things are invisible. At runtime the platform checks that the environment exists in the current consul cluster (if it doesn't there's a problem and it's errors out), it then checks the subtree for the applications namespace (this way there can be no cross contamination where one app references another apps vars) then the version number of the configuration is taken from label connected to the deployable artifact. Updating this label is the key thing here because it means if we lost both production data centres we could stand up the environment again by simple reading the meta data from our deployable artifacts and loading *all the environment variables* into the KV store.

*Problems With This Approach*
Developers always, and I mean every single time, found a way to slip configuration changes into the environment that had significant impacts to how the application would run. Because it always ended up being easier to get configuration changes approved than code changes.

**Storing a "Deployment" Artifact with variables embedded**

This tightly couples the exact version of the artifact to the version of configuration. If you changed configuration then you had to rebuild this deployment artifact.

The deployment artifact itself was essentially a yaml file which contained the URL to the releasable binary and all the configuration attached to it.

The platform contains componentry to read variables and then out them into the process tree of the application when it starts up.

This has so far been a lot more successful because there is an artifact we can trace the history of and which we can hold up to any review board and say "this is the only artifact we care about, we don't need t look at any other changes, only changes to this thing" (ie. Version of application to deploy,  environment variables included etc.

This makes it just a little bit harder for developers to try and build logic into their application that will change its behaviour based on variables so they can slip in changes without going through appropriate testing cycles.

**Bonus Points**

Consider application secrets. Our solution to this has so far been to provide a public RSA key which development teams use to encrypt an extended Java key store (almost every language has a library somewhere that can read Java keys stores) this is then treated like a third type of artifact and is pulled onto the server, decrypted with our platform private key, and provided to the application at run time.

Admittedly secrets management is its own can of worms. But it's probably worth considering.