15

I've been thinking about a Jenkins job to check the pricing for the agent boxes we spin up; every so often the price spikes and it'll be an hour or two before anyone notices that no new agents are coming up, and then we have to go in manually and check the spot price and adjust accordingly, or switch zones, etc.

My first instinct was that the best route is to have an hourly Jenkins job that runs a aws ec2 describe-spot-instance-requests and checks for failed requests (and then slacks us on a failure). But I'm wondering if there's a cleaner method that involves actually comparing hard prices (and could therefore tell us exactly what's off, and by how much), rather than looking at successful/failed requests.

Anyone set up something similar? How'd you do it?

  • 1
    Reasonable question, I guess... but why would you not just bid what the instances are actually worth to you, select multiple instance types (including instances larger than what you need, which are occasionally cheaper when smaller instance spot prices rise... occasionally -- including right now as I write this -- I find myself with a cc2.8xlarge, because it is currently cheaper than the c3.2xlarge I needed, despite having 4x the cores and 4x the memory) and let the system handle instance type selection and AZ placement based on the market? – Michael - sqlbot Apr 18 '17 at 1:23
  • 1
    @Michael-sqlbot I don't have say in that part of the process; we have documentation stating that we purchase a certain size of instance for auditing purposes and so that's what we're stuck with. Just trying to make the most of an unideal situation. – Alex Apr 18 '17 at 12:33
  • 1
    Fair enough, @Alex. Just thought I'd throw it in, in case you hadn't considered it. – Michael - sqlbot Apr 18 '17 at 15:44
  • @Alex Out of interest, is the certain instance size so it can be seen easily on bills? Because you (rather, they!) could use tags and then see them in the Cost Explorer. – Tim Malone Jun 17 '18 at 3:34
15

Spotted an open source tool called autospotting that just might help with this:

Once enabled on an existing on-demand AutoScaling group, it launches an EC2 spot instance that is cheaper, at least as large and configured identically to your current on-demand instances. As soon as the new instance is ready, it is added to the group and an on-demand instance is detached from the group and terminated.

We have this in our to-do pipeline, will be able to add more context once we finish that.

Update:

Another tool that was recently demo'ed in a conference was mapbox/spotswap

This works slightly differently. It monitors a normal ASG with either On-Demand or Reserved instances and then, if scale arises, bids and provisions spot instances of similar compute level on a separate ASG.

  • 2
    Welcome to the site, Hashfyre :) – Dawny33 Apr 17 '17 at 16:52
  • 1
    I'm the author of AutoSpotting, thanks for mentioning it! Did you give it a try yet? I'm all ears for user feedback. – Cristian Măgherușan-Stanciu Oct 15 '17 at 22:10
6

I would personally consider a model like this:

Timed Lambdas -> Checks spot price -> Push to ElastiCache

Then when you need instances:

Timed lambdas -> Pulls spot price from ElastiCache, sets it as environment variable on your Machine where you spin up IaC from -> This is parsed as argument to IaC code and pushes out the spot price

You could set some tolerances, too, within the lambdas (i.e. 10, 25, 50% increases based on importance) and a hard cap of on-demand, for example. It's also a great place to build the logic to handle, for example, finding the cheapest AZ, finding the relatively cheapest spot price (2xt2.medium vs t2.large), etc.

  • Just for information, you could have edited the deleted answer and flagged it to ask for undeletion. (No harm done, it's just informative :)) – Tensibai Apr 19 '17 at 16:16
  • 1
    I will leave my shame to persevere to remind me of my misguided days for all eternity. – Henry Apr 19 '17 at 16:42
  • Rofl, there is really no shame, don't worry :) – Tensibai Apr 19 '17 at 19:02
  • 3
    MY SCARS DEFINE WHO I AM – Henry Apr 19 '17 at 19:04
  • Cool idea - might also be worth mentioning that one could push the price to a DynamoDB table, in case they don't already have an ElastiCache cluster lying around ;). Or maybe even a JSON file in an S3 bucket. – Tim Malone Jun 17 '18 at 3:35
4

Let me give you a tool-agnostic way of trying to do this.

every so often the price spikes and it'll be an hour or two before anyone notices that no new agents are coming up, and then we have to go in manually and check the spot price and adjust accordingly, or switch zones, etc

We faced the same problem in the infrastructure which we're building. So, we had if-else style blocks to set the bid price, depending on the on-demand price of the instance.

AWS has an API for getting the on-demand price of an instance. We used this Python wrapper for the purpose.

So, once we got the on-demand price (let's say X), we plugged in if-else style blocks, which are 0.4*X, 0.6*X, 0.8*X, X, which means we are trying for a bid price in the range 40%, 60%, 80% of the on-demand price. If all fails, then we are falling back to creating on-demand instances.

Also, as this is independent of AWS's current spot pricing, we never pay a price above the on-demand price.

But, if you're looking for a way to do it on-the-fly, then Hashfyre's solution should be the way to go.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.