3

Right now, we only use EC2 on-demand instances, and we provision sensitive data (credentials, private keys, etc.) on the instances' EBS which are encrypted with a KMS key.

My understanding is that I have a few ways to pass those secret data to Spot instances:

  • Burn them in an encrypted AMI (but then, what about host-specific stuff, like SSH server private keys?)
  • Pass them as UserData when we call RunInstances (but then we would need to encrypt those data, and decrypt them with a secret burnt in the AMI)
  • Use Secrets Manager, but that could quickly cost a lot...
  • Use Parameter Store, and encrypt the payload with a KMS key (1 shared key for shared data + 1 key per node for node specific data?)

And I am not sure I could individually identify spot instances?

Am I missing something there? How do you solve those challenges?

Improve this question
  • A variant of instance Ebs is to share an EFS file system – jdog Feb 22 '19 at 6:26
  • 1
    "lot of engineering there" Are you sure? docs.aws.amazon.com/kms/latest/developerguide/… – Michael - sqlbot Feb 22 '19 at 8:03
  • @Michael-sqlbot Thanks for your feedback, I rephrased. – Philippe Feb 22 '19 at 11:51
  • But maybe I am just wrong to think about node-specific data and I should accept that all nodes are the same? It's very hard for me as I cannot, for example, allow myself to blindly trust an unknown SSH host key. – Philippe Feb 22 '19 at 11:52
  • A quick one with the ssh keys, you don't need to trust unknown ssh keys. I've used the approach linked below a lot and it works really well, it's simple to implement and secure enough. Also it's worlds easier to maintain than a key per instance or user. code.fb.com/security/scalable-and-secure-access-with-ssh – Briansbum Feb 22 '19 at 14:02
2

When you're running ephemeral workloads like you would on a spot instance the best bet, if you don't want to bake an ami, is going to be a secret store of some kind.

The two most common methods I've seen for doing this are:

  1. If your provisioning tool (terraform, cloudformation, salt cloud) allows it then you can create a kms key and use it to encrypt secrets in user data. Once the instance has started it can decrypt them using that key.

  2. You can use a secret store like you mentioned. You said that AWS Secrets Manager would get pricey so you're probably going to want to use Vault. It's not that hard to set up and can use IAM permissions to allow your instances access to secrets. Using it also opens other avenues to doing secret storage right.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.