Troubleshooting VPC flow logs with an S3 bucket using SSE-KMS encryption with CMK

Question

I'm using Terraform and trying to set up automatic export of VPC flow logs into an S3 bucket in the same AWS account and region (ca-central-1) that has default encryption turned on with AWS-KMS (using a CMK). Even after trying many permutations of policies for KMS and the S3 bucket, the flow logger still always ends up in Access error status.

Update: When the S3 bucket is reconfigured to use AES-256 as the default encryption (instead of KMS) the VPC flow logs get written normally. So it's definitely a KMS problem.

The aws_flow_log Terraform resource is configured exactly according to the documentation.

resource "aws_flow_log" "vpc_flow_log" {
  log_destination_type = "s3"
  log_destination      = "${aws_s3_bucket.awslogs.arn}"
  traffic_type         = "ALL"
  vpc_id               = "${aws_vpc.vpc.id}"
  depends_on           = ["aws_s3_bucket_policy.awslogs"]
}

And the result of aws ec2 describe-flow-logs:

{
  "CreationTime": "2020-05-15T16:22:40.142Z",
  "DeliverLogsErrorMessage": "Access error",
  "DeliverLogsStatus": "FAILED",
  "FlowLogId": "fl-REDACTED",
  "FlowLogStatus": "ACTIVE",
  "ResourceId": "vpc-REDACTED
  "TrafficType": "ALL",
  "LogDestinationType": "s3",
  "LogDestination": "arn:aws:s3:::REDACTED-awslogs",
  "LogFormat": "${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status}"
}

Curiously, it works fine in my second "sandbox" AWS account where I exclusively use the AWS web console, never Terraform. This account is configured the same way with AWS-KMS on the S3 bucket.

Both accounts seem to have the same configuration, so I can't figure out why it works in the sandbox, but fails in my terraformed account.

---

S3 bucket

Default encryption is enabled and and Custom KMS arn is selected.

S3 bucket policy includes statements to allow VPC flow logs delivery from delivery.logs.amazonaws.com as written in Publishing flow logs to Amazon S3.

    {
        "Sid": "AWSLogDeliveryAclCheck",
        "Effect": "Allow",
        "Principal": {
            "Service": "delivery.logs.amazonaws.com"
        },
        "Action": "s3:GetBucketAcl",
        "Resource": "arn:aws:s3:::REDACTED-awslogs"
    },
    {
        "Sid": "AWSLogDeliveryWrite",
        "Effect": "Allow",
        "Principal": {
            "Service": "delivery.logs.amazonaws.com"
        },
        "Action": "s3:PutObject",
        "Resource": [
            "arn:aws:s3:::REDACTED-awslogs/*",
            "arn:aws:s3:::REDACTED-awslogs/AWSLogs/REDACTED/*"
        ],
        "Condition": {
            "StringEquals": {
                "s3:x-amz-acl": "bucket-owner-full-control"
            }
        }
    }

KMS Key Policy

KMS key policy includes a statement that allows usage by VPC Flow logs as instructed by Required CMK key policy for use with SSE-KMS buckets.

    {
        "Sid": "Allow VPC Flow Logs to use the key",
        "Effect": "Allow",
        "Principal": {
            "Service": "delivery.logs.amazonaws.com"
        },
        "Action": [
            "kms:ReEncrypt",
            "kms:GenerateDataKey",
            "kms:Encrypt",
            "kms:DescribeKey",
            "kms:Decrypt"
        ]
    }

---

I'm at a loss here. What else can I do to troubleshoot this?

Answer 1

Turns out I was missing one very important line in my KMS key policy:

    resources = ["*"]

Now it works fine, and my full policy looks like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Allow VPC Flow Logs to use the key",
            "Effect": "Allow",
            "Principal": {
                "Service": "delivery.logs.amazonaws.com"
            },
            "Action": [
                "kms:ReEncrypt",
                "kms:GenerateDataKey",
                "kms:Encrypt",
                "kms:DescribeKey",
                "kms:Decrypt"
            ],
            "Resource": "*"
        }
    ]
}