From docs.docker.com - force-the-swarm-to-rebalance I can see that Docker Swarm "tasks are given to less busy nodes" to reach eventual balance.
I'm trying to find more info on what "busy" means exactly. For us, running a lot of Java microservices, our primary resource is memory. We are constantly running into OOM killers on the hosts as Swarm keeps scheduling containers to nodes that are already very low on RAM.
I have a 7 nodes swarm in front of me right now with one node's top
stating:
top - 22:55:04 up 234 days, 8:48, 5 users, load average: 0.53, 0.68, 1.09
...
KiB Mem : 18330008 total, 425268 free, 13718032 used, 4186708 buff/cache
KiB Swap: 511996 total, 389836 free, 122160 used. 4225292 avail Mem
This is the lowest in free/available memory in the entire swarm. Yet this node ALWAYS gets the task I'm scheduling. Other nodes are much much more relaxed:
top - 22:56:28 up 5:49, 2 users, load average: 0.38, 0.61, 0.79
...
KiB Mem : 18329940 total, 4899940 free, 7877052 used, 5552948 buff/cache
KiB Swap: 511996 total, 511996 free, 0 used. 10098352 avail Mem
Yet they will never get the task. None of the other nodes will. Only the above mentioned node will, which has by far the lowest available memory.
So what criteria does Swarm use to determine the "business" of a node to determine if a task should be scheduled on it? Available memory clearly does not seem to be a metric in the decision.
Sidenote: I am aware that I can assign resource limits to my containers to prevent the node from being over-scheduled. This is what I am doing right now and I assume it will fix my problem. But I want to know about Swarm's inner workings to determine where to schedule and where not to.