AWS or Azure with usage auto shutdown occasional usage

Question

I'm looking for recommendations on which cloud service provider we should be looking at.

The situation:

I'm part of a small team looking to leverage more powerful analytics. Most of our day to day can be done on local machines but we want to explore using more powerful hardware through a cloud platform before committing to anything in house. As far as software we use Excel, MSSQL, and R, although these cloud instances only need to run R.

Our ideal use case would be uploading data to the cloud, spinning up an instance to run whatever we need to run, then shutting down the instance while keeping the data and software (eg R packages) stored. This way (I think) we would only be charged for the storage while we aren't actively running a project.

Additionally we'd like an auto shutdown feature where the computing instances are automatically shut down if they aren't used for a certain period of time or if that isn't possible, at a given time each day.

We're really outside of our area of expertise here so user friendly is a big plus.

Any advice?

Answer 1

This feature has been available in Azure for a couple of years now, it's as simple as ticking a box and setting a time for shutdown. You can also have it fire a webhook if you need any notifications, or scripts run, or anything else when it shuts down.

Also on Azure there are pre-configured data science VMs and deep learning VMs with all the tools you need including R already installed, you just pick a size and spin one up. You can go small just to try the software, or go for one with multiple GPUs, all already set up with the tools and usable straight away. You can even reboot the same box in different sizes depending on what you need that day!

You are correct that while a VM is not running, you will be charged only for the storage used.

Answer 2

I can think of a few different ways to achieve this.

1. First and easiest way is to create a new CloudWatch Events Rule 1, set its "Event Source" to run on a schedule and "Target" to EC2 StopInstances API call with Instance IDs provided. Look at this screenshot for clarity:

With this example, An event will start at every night at midnight and it will stop Instances by their IDs. You can change the schedule with cron expressions. See: https://crontab.guru

2. You can implement any kind of custom logic with AWS Lambda functions and schedule them to be run with a variety of AWS services (including CloudWatch Events).

For example, with a Lambda function you can implement logic to find instances by a tag, like "GoodBefore=2018-08-10", and stop these instances. In fact you could even make use of API Gateway to expose an API for requesting instances, only to be terminated after a few days by their "GoodBefore" tag.

Of course, while this way is much more customizable and sophisticated, it may prove hard especially if you don't have experience in programming.

3. If you don't want to implement custom logic with Lambda, but also need more than what CloudWatch Events can provide, you can make use of Instance Scheduler solution provided by AWS. It's a similar solution to what I described in previous method, but this time AWS already did the hard work for you.

1 URL for us-east-1 region: https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#rules:action=create

Note: While I haven't used Instance Scheduler myself, I'm pretty sure it wouldn't decide on such things, and I think it only makes sense if it wouldn't do much more than simply making a stop call for the instance. So, on any stop event, either made by an automated solution or you, if your instances' root volume was created "EBS-backed", it will retain the data. If it is "instance store backed", it will lose the data. You can find this information from AWS EC2 console. If you want to retain data after stopping instances, make sure all volumes are EBS-backed when creating a new instance.