Why does the datasource "external" always fail?

Question

so my issue here is that terraform really does not like a command I'm trying to run using an external datasource.

I'm trying to hack my way around a problem where terraform wont wait for a KMS encrypt operation to complete before attempting to use the ciphertext returned by KMS. I've also raised an issue in the terraform for this.

So here is what I'm attempting to do:

resource "aws_kms_alias" "kms_alias" {
    name          = "alias/${var.lambda_name}"
    target_key_id = "${aws_kms_key.hook_url.arn}"
}

data "external" "encrypt_url" {  
    program = ["bash", "export ENC_HOOK_URL=$(aws --profile $${AWS_PROFILE} \
    kms encrypt --key-id $${KMS_ALIAS} --plaintext $${HOOK_URL} --output json \
    --region $${AWS_REGION})"]
}

resource "aws_lambda_function" "lambda_function" {
    depends_on       = ["data.external.encrypt_url"]
    filename         = "deploymentPackage.zip"
    function_name    = "${var.lambda_name}"
    role             = "${var.lambda_role_arn}"
    handler          = "index.lambda_handler"
    source_code_hash = "${base64sha256(file("deploymentPackage.zip"))}"
    runtime          = "python3.6"
    publish          = "true"

    environment {
        variables = {
            HOOK_URL = "${data.external.encrypt_url.result.ciphertextBlob}"
        }
    }

    tags {
        LastUpdated = "${var.timestamp}-${var.iamuser}-Terraform"
    }
}

This will run fine without the encrypt_url datasource but will fail with the following error when trying to run:

data.external.encrypt_url: data.external.encrypt_url: failed to execute "bash":
bash: export ENC_HOOK_URL=$(aws --profile ${AWS_PROFILE} kms encrypt --key-id ${KMS_ALIAS} --plaintext ${HOOK_URL} --output json --region ${AWS_REGION}): 
No such file or directory

Now no file or directory is attempting to be called here and I'm unable to find much information about this because this function is not well documented.

If anyone has any pointers that'd be great!

Answer 1

The command being run by the external provider here is equivalent to the following:

bash "export ENC_HOOK_URL=$(aws --profile ${AWS_PROFILE} \
kms encrypt --key-id ${KMS_ALIAS} --plaintext ${HOOK_URL} --output json \
--region ${AWS_REGION})"

Bash is failing here because when used in this way it expects its first argument to be the filename of a script to run.

The -c option changes this interpretation so that it will instead expect this argument to be an inline script to run, which seems to be what you intended here. This could be expressed in the Terraform configuration like this:

data "external" "encrypt_url" {  
  program = ["bash", "-c", "export ENC_HOOK_URL=$(aws --profile $${AWS_PROFILE} kms encrypt --key-id $${KMS_ALIAS} --plaintext $${HOOK_URL} --output json --region $${AWS_REGION})"]
}

Along with adding the -c option, it also seems like this script doesn't actually do anything, instead just assigning a result to a variable. Given that you asked the AWS command to produce JSON, I think perhaps the following will do what you want:

data "external" "encrypt_url" {  
  program = ["bash", "-c", "aws --profile $${AWS_PROFILE} kms encrypt --key-id $${KMS_ALIAS} --plaintext $${HOOK_URL} --output json --region $${AWS_REGION}"]
}

In the interests of readability though, I'd recommend moving this command into an external script file (e.g. encrypt_url.sh) and then referencing it from the config:

#!/usr/bin/env bash

aws --profile ${AWS_PROFILE} \
    kms encrypt --key-id ${KMS_ALIAS} \
                --plaintext ${HOOK_URL} \
                --output json \
                --region ${AWS_REGION}

...

data "external" "encrypt_url" {  
  program = ["bash", "${path.module}/encrypt_url.sh"]
}

This avoids the need for escaping the dollar signs, allows you to edit it in your text editor's shell script highlighting mode, and allows the script to be easily run outside of Terraform for testing purposes.