I'm looking for recommendations on which cloud service provider we should be looking at.

The situation:

I'm part of a small team looking to leverage more powerful analytics. Most of our day to day can be done on local machines but we want to explore using more powerful hardware through a cloud platform before committing to anything in house. As far as software we use Excel, MSSQL, and R, although these cloud instances only need to run R.

Our ideal use case would be uploading data to the cloud, spinning up an instance to run whatever we need to run, then shutting down the instance while keeping the data and software (eg R packages) stored. This way (I think) we would only be charged for the storage while we aren't actively running a project.

Additionally we'd like an auto shutdown feature where the computing instances are automatically shut down if they aren't used for a certain period of time or if that isn't possible, at a given time each day.

We're really outside of our area of expertise here so user friendly is a big plus.

Any advice?

I can think of a few different ways to achieve this.

  1. First and easiest way is to create a new CloudWatch Events Rule 1, set its "Event Source" to run on a schedule and "Target" to EC2 StopInstances API call with Instance IDs provided. Look at this screenshot for clarity:

Creating a CloudWatch Event

With this example, An event will start at every night at midnight and it will stop Instances by their IDs. You can change the schedule with cron expressions. See: https://crontab.guru

  1. You can implement any kind of custom logic with AWS Lambda functions and schedule them to be run with a variety of AWS services (including CloudWatch Events).

For example, with a Lambda function you can implement logic to find instances by a tag, like "GoodBefore=2018-08-10", and stop these instances. In fact you could even make use of API Gateway to expose an API for requesting instances, only to be terminated after a few days by their "GoodBefore" tag.

Of course, while this way is much more customizable and sophisticated, it may prove hard especially if you don't have experience in programming.

  1. If you don't want to implement custom logic with Lambda, but also need more than what CloudWatch Events can provide, you can make use of Instance Scheduler solution provided by AWS. It's a similar solution to what I described in previous method, but this time AWS already did the hard work for you.

1 URL for us-east-1 region: https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#rules:action=create

  • With the Instance Scheduler solution, does this keep the data separate from the computing instance? That is, when the instance shuts down is my data still retained on the server? – learningasigo Aug 20 at 20:12
  • While I haven't used Instance Scheduler myself, I'm pretty sure it wouldn't decide on such things, and I think it only makes sense if it wouldn't do much more than simply making a stop call for the instance. So, on any stop event, either made by an automated solution or you, if your instances' root volume was created "EBS-backed", it will retain the data. If it is "instance store backed", it will lose the data. You can find this information from AWS EC2 console. If you want to retain data after stopping instances, make sure all volumes are EBS-backed when creating a new instance. – Yekta Leblebici Aug 21 at 0:01

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.