I have an elasticsearch cluster in a kubernetes cluster. I have the data pods going to memory optimized nodes which are tainted so that only the elasticsearch data pods get scheduled to the. Right now I have 3 memory optimized ec2 instances for these data pods. They are r5.2Xlarge's which have 64G of memory. Here is the output of one of these r5 nodes. (they all look the same)
Capacity:
attachable-volumes-aws-ebs: 25
cpu: 8
ephemeral-storage: 32461564Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 65049812Ki
pods: 110
Allocatable:
attachable-volumes-aws-ebs: 25
cpu: 8
ephemeral-storage: 29916577333
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 64947412Ki
pods: 110
System Info:
Machine ID: ec223b5ea23ea6bd5b06e8ed0a733d2d
System UUID: ec223b5e-a23e-a6bd-5b06-e8ed0a733d2d
Boot ID: 798aca5f-d9e1-4c9f-b75d-e16f7ba2d514
Kernel Version: 5.4.0-1024-aws
OS Image: Ubuntu 20.04.1 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://19.3.11
Kubelet Version: v1.18.10
Kube-Proxy Version: v1.18.10
Non-terminated Pods: (5 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
amazon-cloudwatch fluentd-cloudwatch-tzsv4 100m (1%) 0 (0%) 200Mi (0%) 400Mi (0%) 21d
default prometheus-prometheus-node-exporter-tvmd4 100m (1%) 0 (0%) 0 (0%) 0 (0%) 21d
es elasticsearch-data-0 500m (6%) 1 (12%) 8Gi (12%) 8Gi (12%) 14m
kube-system calico-node-dhxg5 100m (1%) 0 (0%) 0 (0%) 0 (0%) 21d
kube-system kube-proxy-ip-10-1-12-115.us-gov-west-1.compute.internal 100m (1%) 0 (0%) 0 (0%) 0 (0%) 21d
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 900m (11%) 1 (12%)
memory 8392Mi (13%) 8592Mi (13%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
attachable-volumes-aws-ebs 0 0
Here is what my cluster looks like
kubectl get pods -n es
NAME READY STATUS RESTARTS AGE
elasticsearch-client-0 1/1 Running 0 77m
elasticsearch-client-1 1/1 Running 0 77m
elasticsearch-data-0 1/1 Running 0 77m
elasticsearch-data-1 1/1 Running 0 77m
elasticsearch-data-2 1/1 Running 0 77m
elasticsearch-data-3 0/1 Pending 0 77m
elasticsearch-data-4 0/1 Pending 0 77m
elasticsearch-data-5 0/1 Pending 0 77m
elasticsearch-data-6 0/1 Pending 0 77m
elasticsearch-data-7 0/1 Pending 0 77m
elasticsearch-master-0 2/2 Running 0 77m
elasticsearch-master-1 2/2 Running 0 77m
prometheus-elasticsearch-exporter-6d6c5d49cf-4w7gc 1/1 Running 0 22h
Here is the events when I describe pod
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 56s (x5 over 3m35s) default-scheduler 0/11 nodes are available: 3 Insufficient memory, 3 node(s) didn't match pod affinity/anti-affinity, 3 node(s) didn't satisfy existing pods anti-affinity rules, 5 node(s) didn't match node selector.
Here is my resource limits and requests for the data pods
Limits:
cpu: 1
memory: 8Gi
Requests:
cpu: 500m
memory: 8Gi
Here is what my nodeAffinity looks like
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: es-data
operator: In
values:
- "true"
And my tolerations
tolerations:
- key: "es-data"
operator: "Equal"
value: "true"
effect: "NoSchedule"
And here is my node tolerance when I describe the node
Taints: es-data=true:NoSchedule
I tainted it like
kubectl taint nodes <node> es-data=true:NoSchedule
According to my calculations based on my understand (which is probably wrong) my data pods are only asking for 8G of memory from a node which has 64G available, and only one pod requesting 8G of memory is already using it. So it should have theoretically 56G of memory left to other pods requesting to be scheduled to it. And even the memory being used shows me it's only 13% used. Why can't it schedule? How can I troubleshoot? Am I misunderstanding how this should work? What else can I tell you which helps troubleshoot this?
Resolution: Based on Hakob's comments, the issue is that I had nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution set, which is a hard requirement directing the scheduler to only schedule one to each node. What I needed to do in order to schedule more than one to each node was change it to nodeSelector. If you are reaching this, please note Hakob's recommendation for why it is not suggested best practice to do this. I agree with that advice. Though in my case it is a requirement coming from client and did not have the option even after have discussion why they should not be doing this. So please take that into consideration when applying this change.
