How to scale-down in a multi-tenant environment?

Question

Cloud environments in AWS allow for multi-tenancy managed by the user himself, classic example are container orchestrators such as ECS or Kubernetes.

When you have two services, one needs memory another cpu and you put these in a single cluster. Then scaling-up is relatively trivial. Each time you need more capacity in terms of either cpu or memory, add more capacity. Since EC2 capacity means units in cpu and memory both.

Scaling up based on a single metric can very easily achieved using CloudWatch Alarms.

When scaling down, in order to reduce cost it requires to take into account both memory and cpu limits and not let any of the two drop below the required amount.

Since unfortunately CloudWatch Alarms do not allow to use boolean logic or take into account multiple metrics.

What is a good way to implement scale down of capacity for an auto scaling group?

Answer 1

Autoscaling is a good case for machine learning

This is a hard problem to do well.

What you really want is something like Nest Thermostat for your EC2 infrastructure.

There are (aforementioned) multiple dimensions of resource demand/limitations.

• CPU
• memory
• disk space
• disk IO
• network IO
• concurrency/latency/queue depth

There are multiple indicators of demand (in addition to the above).

• concurrent unique visits/sessions
• pages per visit/session
• rate of engagement/interaction feature usage

There are multiple common patterns of changing demand over time.

• daily user demand cycles
• weekly user demand cycles
• monthly ...
• annual ...
• special event/days
• DDoS load
• media/marketing exposure traffic spikes

There are multiple financial decision factors.

• does revenue scale with traffic? How? (How conservative do you need to be?)
• is there a hidden cost to control (transaction costs, limits)?
• what's the cost model of scaling? (things in a pipeline scale together, things in a load-balanced cluster scale independently)

Before long, if you try to hand-optimize on hand-selected features you're going to have a monster of technical debt that is possibly more complicated than any other logic in your site. Amazon makes more money when you err (with a large margin) on the side of caution, so their tools will probably never get close to what you want.

Instead, choose an architecture/technology stack that can grow/scale so you don't have to get it exactly right the first time. Then pick a few factors which you think are obvious. Then try to come up with a way to sort multiple representative possibilities in order of preference. Then collect some real world data covering all those points. If you're lucky, a simple obvious hand-coded solution will jump out at you from looking at the data. If not, code up something that will give you an approximate model f(x1,x2,x3,x4) --> y * app nodes, using an appropriate algorithm.

I bet you didn't think this one was going to be so much fun!

Answer 2

To be able to scale in with confidence on ECS, there's a few strategies:

• Create a 3rd custom metric which tracks the greater of the two metrics. For example, if CPU allocation is at 60% and memory allocation is at 70%, the metric should be set to 70%.
• Choose one of the two resources (CPU or memory), and always allocate a higher percentage of it to each container compared to the other resource. This way, you will always reach a shortage of your preferred resource before the other, and you don't have to worry about scaling up or down on it. While this is the easiest solution, the obvious downside is that you can't optimise for services that may be CPU or memory intensive, and you may end up with wasted resources.
• Do autoscaling without using CloudWatch metrics. Tools such as https://github.com/ameir/ECSpander are available to assist with this.

Kubernetes doesn't use CloudWatch metrics for scaling, it manages scaling by setting the ASG desired number of instances through internal mechanisms and therefore isn't subject to the issue you've described.