terraform remote state bucket on s3

Question

I want to use a remote state bucket on s3 for my terraform tfstate files (described here: https://developer.hashicorp.com/terraform/language/settings/backends/s3). It looks pretty basic, but in practice not so straightforward. I tried just defining the bucket like any other s3 resource but terraform won't init/plan anything since the state bucket doesn't exist. So, I used aws-cli to create the state bucket and was planning on dropping all the details into variables.tf but then init produced:

# Error: Variables not allowed

My entire root module/project references variables.tf for everything. From tags, to subnets to AWS region - everything. I've got like 30 resources defined for this project. The path forward that I see here is quite a mess. literal values in the state bucket code, multiple files to maintain the state bucket which are detached from the "project" and probably in a separate git repo, then the bucket and IAM polices and dynamoDB config... ugh.. At this point, the remote state bucket creates more problems than it solves. I'm to the point where a makefile pulling/pushing the .tfstate files back-and-forth to a regular s3 bucket makes more sense. Can someone point me in the right direction here, or is this really the way it is?

state bucket code:

resource "aws_s3_bucket" "my_tf_state_bucket" {
  bucket = var.state_bucket_name  

   tags           = {
      Name              = "${var.my_project}-tf-state-bucket"
      project           = var.my_project
      environment       = var.my_environment
   }

}

terraform {
  backend "s3" {
    bucket         = var.state_bucket_name  # Replace with your S3 bucket name
    key            = "terraform.tfstate"
    region         = var.my_region
    encrypt        = true
    dynamodb_table = "${var.my_tf_state_bucket}-terraform-locking"

  }
}

---

update: In case it's useful for anyone else, and since this was a particularly nasty way of configuring a working state bucket, here is what I ended up doing.

I created s3-state-bucket.tf with the contents below. This allows me to keep the configuration for the state bucket in the same directory as the rest of my terraform files for the project and the commands to create the state bucket can be a shell script or pasted in from comments left in the .tf file. The script can be checked into git/hub/lab with the rest of the project.

Use bash to create a unique bucket name, enable object lock, SSL connections, bucket key + encryption, versioning, block public access

UNIQUE_ID=$((RANDOM % 899 + 100)); MY_BUCKET="S3-uniquely-named-state-bucket-${UNIQUE_ID}"; aws s3api create-bucket --bucket ${MY_BUCKET} --region $AWS_REGION
--create-bucket-configuration LocationConstraint=$AWS_REGION --object-lock-enabled-for-bucket

Enable bucket key and SSE with AWS managed keys to save on costs for KMS key calls

aws s3api put-bucket-encryption --bucket ${MY_BUCKET} --server-side-encryption-configuration '{"Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "AES256"},"BucketKeyEnabled": true}]}'

Enable object lock so tfstate files can't be accidentally modified without explicit locking. Include retention policy for data

aws s3api put-object-lock-configuration --bucket ${MY_BUCKET} --object-lock-configuration '{"ObjectLockEnabled":"Enabled","Rule":{"DefaultRetention":{"Mode":"COMPLIANCE","Days":30}}}'

Ensure public access is specifically disabled, which seems to be an evolving default for AWS but whatever...

aws s3api put-public-access-block --bucket ${MY_BUCKET} --public-access-block-configuration "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

Add a bucket policy to only allow SSL connections to keep compliance happy

aws s3api put-bucket-policy --bucket  ${MY_BUCKET} --policy  "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Deny\",\"Principal\":\"*\",\"Action\":\"*\",\"Resource\":[\"arn:aws:s3:::${MY_BUCKET}\",\"arn:aws:s3:::${MY_BUCKET}/*\"],\"Condition\":{\"Bool\":{\"aws:SecureTransport\":\"false\"}}}]}"

Add tags to the bucket as this aids with cost and resouce managment, monitoring, automation and other processes

aws s3api put-bucket-tagging --bucket ${MY_BUCKET} --tagging 'TagSet=[{Key=environment,Value=production},{Key=project,Value=tf-state-bucket-for-project},{Key=Name,Value='${MY_BUCKET}'}]'

**Added to variables.tf for applying additional access/security controls or configuration **

// the name of the terraform state bucket created using aws-cli so we can apply additional config to the bucket
variable "my_tf_state_bucket" {
  type        = string
  description = "Terraform state bucket created using aws-cli commands"
  default     = "S3-uniquely-named-state-bucket-448"
}

..if you want to remove it all in one shot

aws s3 rm s3://${MY_BUCKET} --recursive; aws s3 rb s3://${MY_BUCKET}

dynamo db table for state locking - not sure if this is optional or not

aws dynamodb create-table --table-name dynamodb-${MY_BUCKET}-locking-table --attribute-definitions AttributeName=LockID,AttributeType=S --key-schema AttributeName=LockID,KeyType=HASH --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

The s3-state-bucket.tf file

#---------------------------------------------------------------
#
# BE SURE TO EDIT THE VALUES BELOW IF RE-CREATING THE BUCKET
# successful tf init contains this:
# $ tf init
#
# Initializing the backend...
# 
# Successfully configured the backend "s3"! Terraform will automatically
# use this backend unless the backend configuration changes.
# Initializing modules...
#
#
# According to `terraform init` "Variables may not be used here."
terraform {
  backend "s3" {
    bucket         = "S3-uniquely-named-state-bucket-448"                        # <----<<< Replace with latest value of ${MY_BUCKET} 
    key            = "terraform.tfstate"
    region         = "us-west-2"                                                 # <----<<< Replace with the region you want... not sure if variable works here
    encrypt        = true
    dynamodb_table = "dynamodb-S3-uniquely-named-state-bucket-448-locking-table" # <----<<< Replace with latest value of ${MY_BUCKET} 

  }
}

Answer 1

A backend block cannot refer to named values (like input variables, locals, or data source attributes). This is from the official documentation Terraform Backend. If you want to use variables in your backend block, you may want to consider using Terragrunt. We personally don't use it but we are considering adding it as there are multiple backends that we need to handle. My advice is don't use variables in the backend as it will not work. If you do not want to add another tool that compliments Terraform, just stick with adding the values to the backend directly.

You can also check this article.

Example:

resource "aws_s3_bucket" "my_tf_state_bucket" {
  bucket = var.state_bucket_name  

   tags           = {
      Name              = "${var.my_project}-tf-state-bucket"
      project           = var.my_project
      environment       = var.my_environment
   }

}

terraform {
  backend "s3" {
    bucket         = s3_bucket_name
    key            = "terraform.tfstate"
    region         = "eu-west-1"
    encrypt        = true
    dynamodb_table = "terraform-locking"

  }
}