16

I've been thinking about a Jenkins job to check the pricing for the agent boxes we spin up; every so often the price spikes and it'll be an hour or two before anyone notices that no new agents are coming up, and then we have to go in manually and check the spot price and adjust accordingly, or switch zones, etc.

My first instinct was that the best route is to have an hourly Jenkins job that runs a aws ec2 describe-spot-instance-requests and checks for failed requests (and then slacks us on a failure). But I'm wondering if there's a cleaner method that involves actually comparing hard prices (and could therefore tell us exactly what's off, and by how much), rather than looking at successful/failed requests.

Anyone set up something similar? How'd you do it?

Improve this question
4
  • 1
    Reasonable question, I guess... but why would you not just bid what the instances are actually worth to you, select multiple instance types (including instances larger than what you need, which are occasionally cheaper when smaller instance spot prices rise... occasionally -- including right now as I write this -- I find myself with a cc2.8xlarge, because it is currently cheaper than the c3.2xlarge I needed, despite having 4x the cores and 4x the memory) and let the system handle instance type selection and AZ placement based on the market?
    – Michael - sqlbot
    Commented Apr 18, 2017 at 1:23
  • 1
    @Michael-sqlbot I don't have say in that part of the process; we have documentation stating that we purchase a certain size of instance for auditing purposes and so that's what we're stuck with. Just trying to make the most of an unideal situation.
    – Alex
    Commented Apr 18, 2017 at 12:33
  • 1
    Fair enough, @Alex. Just thought I'd throw it in, in case you hadn't considered it.
    – Michael - sqlbot
    Commented Apr 18, 2017 at 15:44
  • @Alex Out of interest, is the certain instance size so it can be seen easily on bills? Because you (rather, they!) could use tags and then see them in the Cost Explorer.
    – Tim Malone
    Commented Jun 17, 2018 at 3:34

3 Answers 3

Reset to default
15

Spotted an open source tool called autospotting that just might help with this:

Once enabled on an existing on-demand AutoScaling group, it launches an EC2 spot instance that is cheaper, at least as large and configured identically to your current on-demand instances. As soon as the new instance is ready, it is added to the group and an on-demand instance is detached from the group and terminated.

We have this in our to-do pipeline, will be able to add more context once we finish that.

Update:

Another tool that was recently demo'ed in a conference was mapbox/spotswap

This works slightly differently. It monitors a normal ASG with either On-Demand or Reserved instances and then, if scale arises, bids and provisions spot instances of similar compute level on a separate ASG.

Improve this answer
2
  • 2
    Welcome to the site, Hashfyre :)
    – Dawny33
    Commented Apr 17, 2017 at 16:52
  • 1
    I'm the author of AutoSpotting, thanks for mentioning it! Did you give it a try yet? I'm all ears for user feedback.
    – Cristian Măgherușan-Stanciu
    Commented Oct 15, 2017 at 22:10
6

I would personally consider a model like this:

Timed Lambdas -> Checks spot price -> Push to ElastiCache

Then when you need instances:

Timed lambdas -> Pulls spot price from ElastiCache, sets it as environment variable on your Machine where you spin up IaC from -> This is parsed as argument to IaC code and pushes out the spot price

You could set some tolerances, too, within the lambdas (i.e. 10, 25, 50% increases based on importance) and a hard cap of on-demand, for example. It's also a great place to build the logic to handle, for example, finding the cheapest AZ, finding the relatively cheapest spot price (2xt2.medium vs t2.large), etc.

Improve this answer
5
  • Just for information, you could have edited the deleted answer and flagged it to ask for undeletion. (No harm done, it's just informative :))
    – Tensibai
    Commented Apr 19, 2017 at 16:16
  • 1
    I will leave my shame to persevere to remind me of my misguided days for all eternity.
    – Henry
    Commented Apr 19, 2017 at 16:42
  • Rofl, there is really no shame, don't worry :)
    – Tensibai
    Commented Apr 19, 2017 at 19:02
  • 3
    MY SCARS DEFINE WHO I AM
    – Henry
    Commented Apr 19, 2017 at 19:04
  • Cool idea - might also be worth mentioning that one could push the price to a DynamoDB table, in case they don't already have an ElastiCache cluster lying around ;). Or maybe even a JSON file in an S3 bucket.
    – Tim Malone
    Commented Jun 17, 2018 at 3:35
4

Let me give you a tool-agnostic way of trying to do this.

every so often the price spikes and it'll be an hour or two before anyone notices that no new agents are coming up, and then we have to go in manually and check the spot price and adjust accordingly, or switch zones, etc

We faced the same problem in the infrastructure which we're building. So, we had if-else style blocks to set the bid price, depending on the on-demand price of the instance.

AWS has an API for getting the on-demand price of an instance. We used this Python wrapper for the purpose.

So, once we got the on-demand price (let's say X), we plugged in if-else style blocks, which are 0.4*X, 0.6*X, 0.8*X, X, which means we are trying for a bid price in the range 40%, 60%, 80% of the on-demand price. If all fails, then we are falling back to creating on-demand instances.

Also, as this is independent of AWS's current spot pricing, we never pay a price above the on-demand price.

But, if you're looking for a way to do it on-the-fly, then Hashfyre's solution should be the way to go.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.