Continuous Deployment best practices for managing application configuration

Question

My CD pipeline:

1. Get Docker image (created and pushed during CI stage).
2. Copy docker-compose .yml template to target server.
3. Run the app.

Let's go straight to the 3rd point. I can run the app using docker-compose up e.g., but I have no idea how to handle configuration. Target server is a Ubuntu Server created by Chief on AWS, I have an access to all AWS services like Parameter Store etc.

So here are my ideas about managing configurations:

1. save secrets to configuration file before running the app and attach this file to specific location in docker-compose template (I don't like this solution, I can handle this by some bash script that inject secrets to file).
2. set environment variables before running the app (I talked with some DevOps and basically they like this solution).
3. get secrets from AWS Parameter Store (unfortunately I can't have an access by role management, only by key which I need to set on server by env or in config file).

How does managing configuration look like in real case scenarios? I'm worried about case of server crash or something similar. 1st solution will survive this, but in case of 2nd and 3rd ones the environment variables will be lost after crash/reboot. What is the best solution here?

Answer 1

[Copying answer over from duplicate post]

If you start a container with an environment variable, that variable is part of the container definition. As a pro, when the server is rebooted, the same variable will be there when the container restarts. But as a con, anyone with access to the docker API can view that secret in clear text, and some apps log their environment when debugging.

With swarm mode (Kubernetes has their own similar solution), you can define your secret on the manager and it will be given to nodes running the container on a need to know basis. The secret is accessible inside the container as a file, but that file is only stored in memory on the worker node, and you can use an entrypoint script in your container to load that into the environment if that's how the application expects it. The swarm mode secrets and configs have the advantage of being portable as the container is rescheduled on other nodes in the event of a worker crash. Migrating from a docker-compose install to a single node swarm cluster and deploying a stack is a very simple task.

There are also external solutions for storing configs and secrets in key/value stores like CoreOS's etcd and Hashicorp's Vault. Both Swarm Mode and Kubernetes use a key/value store for their own settings, giving you an idea of how the community sees it as a best practice. The downside is you often want to configure the application to read and write configuration from one of these key/value stores directly. If you use something like swarm mode secrets and configs, these will be stored in docker's internal key/value database, so you get a similar solution with little effort on your part.

Answer 2

Our scenario is very similar to yours. Our configuration is a plaintext YAML file that is stored on the host that is bound mounted into the container.

It's not ideal because secrets are in plaintext. It's also problematic to maintain parity between 2 nodes.

As someone mentioned in the comments; k8s is one potential solution, but it feels like an overkill for a 2 node system. I haven't found anything between docker-compose and k8s yet.

Answer 3

I guess the point here is that you want to keep the deployment approach the same across all environments and update configuration in one place.

If you have had a look at the 12 factor app it states application config should live in the environment e.g. vault, config store, AWS Parameter Store. Kubernetes secrets scoped to a namespace.

Environment variables are also not considered very secure as they can be inspected at runtime.

I prefer option 3 for a number of reasons:

1. Application would pull configuration from config store at runtime (AWS Parameter store).
2. Application config can be changed dynamically (no need to re-deploy the app)
3. No need to add logic to your deployment scripts that pulls secrets from a config store and injects then into the compose file during deployment.
4. Your deployment orchestration tool does not need access to your applications secrets.

On the other hand:

1. Application will need to be updated to support pulling config from a store. This might make the application not cloud agnostic.

On a side not (not related to your question), if you're just deploying a few containers for a simple application and k8 is not suitable why don't you have a look at ECS Fargate? No need to manage any underlying infrastructure.