When your micro service in production needs a secret, where is the secret stored and retrieved from?

Question

The context of the question is Secrets management.

How do you manage storing and retrieving secrets for your micro service which is in production?

Answer 1

Short answer

Ideally, you should store secrets as environment variables, and retrieve them from a secrets management system like Hashicorp's Vault or AWS Parameter store.

Long answer

I saw your questions out of turn, and kinda touched on this in your other question:

Again, there are many perfectly valid options for handling secrets: Chef vault, 1password, environment variables injected by your build instances, home-grown solution using encrypted files stored on object storage, etc....

It really depends on how you run and deploy your apps. I worked at a company that used Amazon Beanstalk for production, and they shipped all their secrets and configs as a 'secrets.conf' file that was created by the Jenkins build server, and shipped with the app when it deployed. The problem there was that it was unruly to manage the same secrets across multiple jobs in Jenkins. So, I came in and moved them towards using Chef to create the file instead, and Chef pulled the file from Hashicorp Vault.

At my current company, they are heavy users of GitLab with it's built-in CI/CD build system. Because the the apps are all Docker instances running in Kubernetes, all secrets are actually set as environment variables set during the deployment. The build server gets those from 'secret variables' that is set from the GitLab UI.

These fit the business need at the time, and that's what really matters at the end of the day. However, you if want to be proper, you should store secrets as environment variables because you specifically mention micro services. It is highly recommend that you follow the 12 Factor App approach to configs: https://12factor.net/config

The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally; and unlike custom config files, or other config mechanisms such as Java System Properties, they are a language- and OS-agnostic standard.

So you'll definitely want to leverage environment variables. How you set those environment variables could be your Configuration Management system (Ex: Chef), or your build server when it does the deploy, or your apps could be coded to pull directly from your secret storage and set the variables internally.

For actual place where the secrets live (where they are retrieved from), you have a few options. But the 2 you mentioned in your comment would be really great ones to use: Hashicorp's Vault and AWS Parameter store. Others would be Chef Vault, home-grown solution using encrypted files stored on object storage, 1password, etc.... Ideally, you'd want something that can be programmatically access from your build pipeline or apps.

Answer 2

We use Chef Vault at my company. Here's a quick tutorial How to Create Chef-Vault

Answer 3

Take a look at https://pkhub.io its a cloud secrets manager that's easy to setup and use, see https://pkhub.io/usecases/environments, for a python example.