Defining and using bash functions in AWS CodeBuild

Question

I thought I could slightly improve readability of repetitive sed calls when inserting secrets into a file with AWS CodeBuild. I defined a bash function:

replaceConfig() { sed -i 's|{'$1'}|'${!1}'|' config.json; }

This can be called like so:

replaceConfig DB_PASSWORD

And this will replace all instances of the text {DB_PASSWORD} in config.json with the value held in secret environment variable DB_PASSWORD. Unfotunately, this does not work in CodeBuild. Inspecting the log file, I see:

[Container] 2020*07*31 15:44:40 Running command replaceConfig DB_USERNAME
*codebuild*output*tmp*script.sh: line 4: replaceConfig: command not found

I get the same error defining the bash function with the alternative syntax:

function replaceConfig { sed -i 's|{'$1'}|'${!1}'|' config.json; }

I have read that buildspec version 0.1 used a separate shell process for each command, which would have caused this problem, but I am using buildspec version 0.2, so these commands should be sharing the same shell.

Of course I can just stick with the repetitive sed calls, but am I correct in understanding that bash functions cannot be used in buildspec commands at all? Is there a better way to get these secrets into a configuration file?

version: 0.2

run-as: root

env:
  shell: bash
  variables:
    AWS_REGION: us-east-2
  secrets-manager:
    DB_USERNAME: db-credentials:username
    DB_PASSWORD: db-credentials:password
    DB_HOST: db-credentials:host
....
phases:
  ....
  post_build:
    commands:
      - replaceConfig() { sed -i 's|{'$1'}|'${!1}'|' config.json; }
      - replaceConfig DB_USERNAME
      - replaceConfig DB_PASSWORD
      - replaceConfig DB_HOST
....

Answer 1

Bash functions (even when exported) are not preserved across commands steps. This is in contrast to exported env variables, so it seems AWS is doing some shenanigans behind the scenes to prevent it. I see two basic options to work around that:

1. Combine steps which need the function in a single step. E.g.

   commands:
     - |
       replaceConfig() { sed -i 's|{'$1'}|'${!1}'|' config.json; }
       replaceConfig DB_USERNAME &&
         replaceConfig DB_PASSWORD &&
         replaceConfig DB_HOST
   

2. Source the file with the functions you need in every step. This may sound laborious, but can be automated via the BASH_ENV environment variable. E.g. let's say you put all the functions you need in a codbuild-helper.bash file in your project root, then you can do:

   env:
     shell: bash
     variables:
       AWS_REGION: us-east-2
       BASH_ENV: "$CODEBUILD_SRC_DIR/codebuild-helper.bash"
     secrets-manager:
       DB_USERNAME: db-credentials:username
       DB_PASSWORD: db-credentials:password
       DB_HOST: db-credentials:host
   ....
   phases:
     ....
     post_build:
       commands:
         - replaceConfig DB_USERNAME
         - replaceConfig DB_PASSWORD
         - replaceConfig DB_HOST
   ....
   

   But since this file will be sourced in every step (regardless if it's needed or not) and even in bash scripts called along the way, you should take care to indeed only put function definitions in there.

Answer 2

In cases where you have a complex bash script with functions for AWS CodeBuild that may not translate well into YAML syntax, encoding the script with base64 provides a robust solution. It preserves the script's integrity, particularly when it contains special characters or constructs that are problematic to escape directly in YAML.

Here's a streamlined approach

import base64

# Read the script content from a file
with open('path/to/your/script.sh', 'r') as file:
    script_content = file.read()

# Base64 encode the script to avoid issues with special characters
encoded_script = base64.b64encode(script_content.encode()).decode()

# Generate the commands to decode and execute the script within the build environment
commands = [
    "echo 'Starting the pre-build process'",
    f"echo {encoded_script} | base64 --decode > setup_env.sh",
    "chmod +x setup_env.sh",
    "./setup_env.sh"
]

# Integrate these commands into your buildspec configuration
build_spec_yaml = {
    "version": "0.2",
    "env": {
        "parameter-store": {
            "ModelPackageName": config['SsmModelPackageNameParameter'],
        }
    },
    "phases": {
        "pre_build": {
            "commands": commands
        }
        # Other phases as needed...
    }
    # Additional buildspec configurations...
}

# If using AWS CDK, you can then convert this dictionary to YAML format
build_spec = BuildSpec.from_object_to_yaml(build_spec_yaml)

This method is advantageous for those utilizing AWS CDK for their infrastructure as code, allowing the dynamic generation and modification of buildspec configurations directly from Python scripts. It leverages the BuildSpec.from_object_to_yaml method for seamless translation to YAML, which can then be used by CodeBuild for provisioning through AWS CloudFormation.

Hope this solution aids anyone facing similar challenges and enhances the flexibility of managing build specifications in complex environments.

Feel free to share your insights or improvements to this approach!