7

I thought I could slightly improve readability of repetitive sed calls when inserting secrets into a file with AWS CodeBuild. I defined a bash function:

replaceConfig() { sed -i 's|{'$1'}|'${!1}'|' config.json; }

This can be called like so:

replaceConfig DB_PASSWORD

And this will replace all instances of the text {DB_PASSWORD} in config.json with the value held in secret environment variable DB_PASSWORD. Unfotunately, this does not work in CodeBuild. Inspecting the log file, I see:

[Container] 2020*07*31 15:44:40 Running command replaceConfig DB_USERNAME
*codebuild*output*tmp*script.sh: line 4: replaceConfig: command not found

I get the same error defining the bash function with the alternative syntax:

function replaceConfig { sed -i 's|{'$1'}|'${!1}'|' config.json; }

I have read that buildspec version 0.1 used a separate shell process for each command, which would have caused this problem, but I am using buildspec version 0.2, so these commands should be sharing the same shell.

Of course I can just stick with the repetitive sed calls, but am I correct in understanding that bash functions cannot be used in buildspec commands at all? Is there a better way to get these secrets into a configuration file?

version: 0.2

run-as: root

env:
  shell: bash
  variables:
    AWS_REGION: us-east-2
  secrets-manager:
    DB_USERNAME: db-credentials:username
    DB_PASSWORD: db-credentials:password
    DB_HOST: db-credentials:host
....
phases:
  ....
  post_build:
    commands:
      - replaceConfig() { sed -i 's|{'$1'}|'${!1}'|' config.json; }
      - replaceConfig DB_USERNAME
      - replaceConfig DB_PASSWORD
      - replaceConfig DB_HOST
....

4
  • Welcome to DevOps.SE, Kyle. This seems like a good idea to me. I'm not sure why it doesn't work. Have you tried contacting AWS support?
    – chicks
    Aug 3, 2020 at 22:14
  • Thank you for commenting. I use free tier, which as far as I know does not allow me to contact AWS support for this kind of issue. I will try posting in the AWS Developer Forums though.
    – Kyle
    Aug 15, 2020 at 15:35
  • 1
    Sidenote: using envsubst is easier than sed almost every time. envsubst < config.json.tpl > config.json. It substitutes env vars into a template file, simple one liner. We can also tell envsubst to only replace a set of specific var names if desired. envsubst is installed by default since codebuild standard 2.0 iirc Aug 4, 2021 at 18:21
  • @OnethingSimple Thanks for this. Although it's a sidenote to the "XY Problem" I asked about, it does solve my original problem. :-)
    – Kyle
    Jan 17, 2022 at 14:57

1 Answer 1

1

Bash functions (even when exported) are not preserved across commands steps. This is in contrast to exported env variables, so it seems AWS is doing some shenanigans behind the scenes to prevent it. I see two basic options to work around that:

  1. Combine steps which need the function in a single step. E.g.

    commands:
      - |
        replaceConfig() { sed -i 's|{'$1'}|'${!1}'|' config.json; }
        replaceConfig DB_USERNAME &&
          replaceConfig DB_PASSWORD &&
          replaceConfig DB_HOST
    
  2. Source the file with the functions you need in every step. This may sound laborious, but can be automated via the BASH_ENV environment variable. E.g. let's say you put all the functions you need in a codbuild-helper.bash file in your project root, then you can do:

    env:
      shell: bash
      variables:
        AWS_REGION: us-east-2
        BASH_ENV: "$CODEBUILD_SRC_DIR/codebuild-helper.bash"
      secrets-manager:
        DB_USERNAME: db-credentials:username
        DB_PASSWORD: db-credentials:password
        DB_HOST: db-credentials:host
    ....
    phases:
      ....
      post_build:
        commands:
          - replaceConfig DB_USERNAME
          - replaceConfig DB_PASSWORD
          - replaceConfig DB_HOST
    ....
    

    But since this file will be sourced in every step (regardless if it's needed or not) and even in bash scripts called along the way, you should take care to indeed only put function definitions in there.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.