Terraform: add to existing AWS policy, or create policy if needed

Question

Several of our terraform root modules need add to an existing policy that provides read-only permissions for S3 buckets -- each module has its own bucket. I cannot figure out how to add to the existing policy, and merge new values (the new bucket) into it.

At first, we had just attached a new policy (three actually, one for development, staging, production) to the role. But we can only attach a maximum of 20 policies. So instead, I used the AWS CLI and just listed all the production buckets in one policy, staging, and dev likewise. Cool so far.

Now I want to accomplish the same thing with terraform.

I have created a submodule that the root modules can all call, passing their bucket in.

I can define the new policy as

data "aws_iam_policy_document" "read_only" {
  statement {
    actions = [
      "s3:ListBucket",
      "s3:GetObject",
      "s3:GetBucketLocation",
    ]
    effect = "Allow"
    resources = [
      "${var.bucket_arn}/*",
      var.bucket_arn,
    ]
  }
}

I can look up an existing policy with

data "aws_iam_policy" "existing" {
  name = "read-only-s3-${terraform.workspace}"
}

and potentially merge that with the new policy using

data "aws_iam_policy_document" "merged" {
  source_json = [
    data.aws_iam_policy_document.read_only.json,
    data.aws_iam_policy.existing.json
  ]
}

and then create the resource

resource "aws_iam_policy" "read_only_policy" {
  name        = "read-only-s3-${terraform.workspace}"
  path        = "/"

  policy = data.aws_iam_policy_document.merged.json
}

But this fails if there's no data.aws_iam_policy.existing because AWS cannot find it so terraform fails with an error.

There's a great deal of documentation about how to merge policies in the policy document reference, but there's very little indication of how to determine if there's an existing policy.

I could recreate the entire policy if I could find the existing one and merge it with the new one, or even if I just knew the names of the buckets in the existing policy. But the existing policies are created in other root modules. And there's no way to look up buckets other than by their name (no tags or filter on data "aws_s3_bucket").

Is there a way to determine that there's an existing policy and get without throwing an error? Then I could get its policy document and use the source_json approach.

Is there a way that I can merge a new policy document onto whatever exists (or not)?

Open to other approaches. Really would prefer to keep this in IAM and not go to bucket policies because they are so opaque and decentralized.

Answer 1

This is not a good way to go about this, terraform is a declarative language and you should always follow this principle.

Either terraform is managing a resource or terraform is querying an existing resource in order to provide information to manage a resource. If you allow scripting, very soon people will be implementing OOP, interaces, generics and it loses the whole focus.

With that out of the way, yes, you can do this. What you need is the policy document, in that case, you can use some non standard strategies like an external data source

the external data source will return a string with instructions for what to do next. If the policy exists, it returns "yes" if it doesn't it returns "nope"

data "external" "iam-policy-exists" {
  program = ["bash", "${path.module}/iam-policy-exists.sh"]

  query = {
    policy = "read-only-s3-${terraform.workspace}"
  }
}

then, the bash script that will check if the iam policy exists

iam-policy-exists.sh

#! /bin/bash

aws iam get-policy $1 #tries to fetch the policy

if $? == "0"; then #$? returns the result of the last command, 0 is success
    echo '{"exists":"yes"}'
else
    echo '{"exists":"nope"}'
fi

then you have a data query with a count

data "aws_iam_policy" "existing" {
  #this count == 1 if the policy exists
  count = data.external.iam-policy-exists.exists == "yes" ? 1 : 0
  name = "read-only-s3-${terraform.workspace}"
}

resource "aws_iam_policy" "read_only_policy" {
  #this will be 1 if the policy doesn't exists
  count = data.external.iam-policy-exists.exists == "nope" ? 1 : 0
  name        = "read-only-s3-${terraform.workspace}"
  path        = "/"

  policy = data.aws_iam_policy_document.merged.json
}

So far, this is what you wanted, to either query or create an IAM policy.

Now, if you want to create a second merged policy you can get the resulting json

locals {
    iam_policy_json = data.external.iam-policy-exists.exists == "yes" ? data.aws_iam_policy.existing[0].json : resource.aws_iam_policy.read_only_policy[0].json
}

and now, you have the json policy on local.iam_policy_json

Disclaimer, see how ugly things get when you try to do the procedural way? I'm not trying to compel you to do this differently, but it would be a good idea to rethink your approach to this.