How can I manage secrets in .tf and .tfstate?

Question

I would like to use the Terraform MySQL Provider to keep a list of mysql users and grants handy for creating new test environments. The .tf and .tfstate files both seem to want to store the MySQL passwords in plaintext.

Concerning .tf:

It is my understanding that .tf files live in revision control and are maintained by a team. How does that practice differ when secrets are in the .tf? It is possible to encrypt these values at all?

Concerning .tfstate:

I can store the .tfstate securely somewhere after running Terraform apply, but it would be preferable for this use case to not store it at all?

Answer 1

Terraform supports adding an additional file with variables during invocation.

^{documentation: https://www.terraform.io/intro/getting-started/variables.html#from-a-file}

We are using that feature to provide a secrets.tfvars file on each invocation of Terraform. We also use a script to wrap the command so that its invocation is consistent, and all team members avoid having to make the same mistakes. The wrapper synchronizes .tfstate with S3 before an execution, and pushes .tfstate back to S3 at the end. I also hear of people doing the same thing with state stored in Consul, even adding a kind of semaphore in consul to prevent two people from starting Terraform at the same time.

When you avoid setting a default value in a variables.tf file, it forces the user to input the value. It can be either entered manually or using the -var-file command option like described above. Not setting a default on your secrets is a good way to enforce changes that require a change in secrets.

The secrets.tfvars file is a symbolic link to one of the files with secrets which are not stored in version control. We have several, one per environment, like so secrets-prod.tfvars, secrets-dev.tfvars, secrets-stg.tfvars, etc...

An even better practice would be to generate these secrets files during the wrapper script based on data in Vault or some other way to share secrets. Since currently when the format of secrets changes, or secrets themselves, we need to communicate it to the team outside the version control channel - and this doesn't always work well, to be honest. But secrets do change infrequently.

Answer 2

We avoid terraform handle our secrets. Even if you manage to inject secrets by a var file "secrtes.tfvars" as pointed out above, these secrets will be stored in your terraform (remote-)state.

You can protect remote-state files by using e.g. S3 authorization, or you can gitignore local state files but we decided not to rely on this kind of protection.

Answer 3

If you're on AWS, then have a look at "The Right Way to Manage Secrets" by Segment.io on the AWS Blog. We advocate using chamber to all of our customers for managing secrets. It works by leveraging the AWS Systems Manager Parameter Store (SSM) together with KMS keys. This ensures secrets are encrypted at rest (and in transit), secured with IAM, auditable with CloudTrails, and only exposed as environment variables at run-time.

After configuring chamber and setting up the KMS key, we write the secrets to the parameter store.

chamber write db TF_VAR_DB_USER foobar
chamber write db TF_VAR_DB_PASS secret

Then use those secrets when you call terraform.

chamber exec db -- terraform plan

This assumes you've defined a variable called DB_USER and DB_PASS in your HCL code.

For example, you could add this to variables.tf

variable "DB_USER" { }
variable "DB_PASS" { }

NOTE: chamber will always export environment variables in uppercase

We provide a terraform module called terraform-aws-kms-key to make provisioning the KMS key easy. Check out our detailed documentation with examples of how to use chamber with multiple namespaces as well as how to use chamber with terraform to manage secrets. See our complete reference example for provisioning chamber dependencies.

As for .tfstate, you bring up a really good point about the existence of plain-text secrets in the state file. There's really no way around this. In order for terraform to calculate changes to build a plan, it needs to know the "before" and "after" state. For this reason, we recommend using an encrypted S3 bucket with mandatory versioning. Use the terraform-aws-tfstate-backend module to provision a bucket and DynamoDB locking table according to best practices.

Answer 4

To import secrets into .tf files, you can also use an external data source. This could be e.g. a script which decrypts your secrets.

Answer 5

I looked at a few different ways but I particularly liked git-crypt for an adhoc thing to do before implementing something bigger like Vault.

Answer 6

You can utilize the external data source to import secrets to Terraform and encrypt the secret files with gpg.

For instance, this what a terraform file would like for creating an rds instance with password foobarbaz

data "external" "rds" {
  program  = [ "cat", ".secrets/rds.json"]
}


resource "aws_db_instance" "default" {
  allocated_storage    = 20
  storage_type         = "gp2"
  engine               = "mysql"
  engine_version       = "5.7"
  instance_class       = "db.t2.micro"
  name                 = "mydb"
  username             = "foo"
  password             = "${data.external.rds.result.password}"
  parameter_group_name = "default.mysql5.7"
}

You can check this post that explains in detail how to do this.

How to encrypt terraform secrets