I do R&D work on spot GPU's on AWS and have been, so far, launching them from a 50GB AMI that I create every time I need to shut off the instance. This has been working pretty well, except that startup times/accessing data on the AMI takes a while at first access because the AMI has to be downloaded from S3 and unpacked (I think this is how it works).

The reason I need to persist the image is that I want my configuration for python packages, as well as the packages themselves to be persisted, and for data to be saved locally in an uncompressed format. So, I can't just mount an NFS to solve this problem (although all of my data/code resides on an NFS in some format, and I just move the data over to the EBS once launched to decompress it.

For some reason EFS is slow with dealing with lots of small files, but thats a separate issue). It seems that EC2 doesn't allow using an EBS volume as root volume for spot instances.

Is there a way to achieve the above mentioned goals without purchasing the on-demand instance?

  • What is the NFS client version of your AMI ?
    – Tensibai
    Sep 19 '17 at 7:38
  • Why does your data need to be stored in an uncompressed format? Why is EFS being slow a separate issue - if it was fast, would you use it? Sep 19 '17 at 8:29
  • If EFS was fast, then I could store all of my uncompressed data on it, but I would still need a way to quickly access already installed/configured packages. The reason the data needs to stay uncompressed is that I don't want to have to copy it over to a readable format for image processing every time I start up the instance. Sep 23 '17 at 5:10
  • NFS in general is subject to consistent latency, even under NFS 4.1 (though it is better than 4.0). A series of small files is actually not NFS' strong suit. What is your AMI based off of? Also, what Instance Type are you using?
    – Matt O.
    Sep 23 '17 at 18:48
  • I've used NFS successfully with small files before. This is a problem inherent to EFS, not NFS in general. To give you an idea, copying over 1000 small files (10KB each) from one folder on an EFS to another folder on the same EFS could take 5 minutes. My AMI is an Ubuntu 14.04 base image that I installed a bunch of python packages that I'd like to keep. I know that I could probably use docker, but the problem with that is I'd still have to run the docker image on first boot which would mean a 3-5 min install time. Sep 23 '17 at 19:10