How does Docker Swarm balance tasks?

Question

From docs.docker.com - force-the-swarm-to-rebalance I can see that Docker Swarm "tasks are given to less busy nodes" to reach eventual balance.

I'm trying to find more info on what "busy" means exactly. For us, running a lot of Java microservices, our primary resource is memory. We are constantly running into OOM killers on the hosts as Swarm keeps scheduling containers to nodes that are already very low on RAM.

I have a 7 nodes swarm in front of me right now with one node's top stating:

top - 22:55:04 up 234 days,  8:48,  5 users,  load average: 0.53, 0.68, 1.09
...
KiB Mem : 18330008 total,   425268 free, 13718032 used,  4186708 buff/cache
KiB Swap:   511996 total,   389836 free,   122160 used.  4225292 avail Mem

This is the lowest in free/available memory in the entire swarm. Yet this node ALWAYS gets the task I'm scheduling. Other nodes are much much more relaxed:

top - 22:56:28 up  5:49,  2 users,  load average: 0.38, 0.61, 0.79
...
KiB Mem : 18329940 total,  4899940 free,  7877052 used,  5552948 buff/cache
KiB Swap:   511996 total,   511996 free,        0 used. 10098352 avail Mem

Yet they will never get the task. None of the other nodes will. Only the above mentioned node will, which has by far the lowest available memory.

So what criteria does Swarm use to determine the "business" of a node to determine if a task should be scheduled on it? Available memory clearly does not seem to be a metric in the decision.

Sidenote: I am aware that I can assign resource limits to my containers to prevent the node from being over-scheduled. This is what I am doing right now and I assume it will fix my problem. But I want to know about Swarm's inner workings to determine where to schedule and where not to.

Answer 1

AFAIK, Swarm does zero real-time resource monitoring. It's only as smart as reservations and limits that you set on the Service beforehand.

First, it takes into account if the tasks of that Service are spread out. It will try not to schedule multiple tasks on the same node, and may ignore total task count per node to do so.

Then, it counts the number of tasks per node and finds the one with the least task count.

All these is restricted by resource reservations. If that node doesn't have the room (based only on the reservations you set), it'll look to the next lower task count.

This is also why Swarm alone can't do built-in auto-scaling because it has no way to track metrics over time.

It does know total CPU and memory, so it knows how to track "full" once you've set resource reservations.

More info: https://github.com/docker/swarmkit/blob/master/design/scheduler.md