Unable to send cloudwatch logs to loggly using a blueprint lambda - Invalid ciphertext

Question

I am trying to use the loggly Lambda blueprint to send cloudwatch logs to loggly. I am doing this because I have a bunch of lambdas that I need to have a single point of viewing logs. I've followed this guide and have run into an issue when I check the lambda logs after a test

Command used to do ciphertext

aws kms encrypt --key-id alias/logglyCustomerToken --plaintext "<my token>"

KMS key was created using symetric and default settings, can't delete key to try again or at least I don't know how to delete it and thinking I shouldn't change the code to match a new one

Lambda log message when failed test

INFO    InvalidCiphertextException: null
    at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:51:27)
    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'InvalidCiphertextException',
  time: 2020-07-08T02:48:55.536Z,
  requestId: 'cf0df165-5a59-4db0-beef-5d1ae3619c37',
  statusCode: 400,
  retryable: false,
  retryDelay: 79.27899980360486
}

Answer 1

Follow these steps to fix InvalidCiphertextException:

1. Open the Lambda function in AWS console.

2. Scroll down to the Environment Variables section & click Edit:

3. Delete the value of kmsEncryptedCustomerToken & replace it with the customer token you got from https://<your-company>.loggly.com/tokens. The token looks like this: 72cf6d64-256e-449d-aabd-49e1f422ed58.

4. Expand the Encryption Configuration section at the bottom.

5. Select Enable helpers for encryption in transit.

6. Click the Encrypt button that appears next to kmsEncryptedCustomerToken.

7. Select the logglyCustomerToken KMS key (CMK) & Encrypt.

8. Finally, Save the environment variables.

The Lambda function test should succeed now. If you face any other issues or need a complete end-to-end screenshot-guided tutorial for the entire process of sending CloudWatch Logs to Loggly, please see my blog post.