1

I have an elasticsearch cluster in a kubernetes cluster. I have the data pods going to memory optimized nodes which are tainted so that only the elasticsearch data pods get scheduled to the. Right now I have 3 memory optimized ec2 instances for these data pods. They are r5.2Xlarge's which have 64G of memory. Here is the output of one of these r5 nodes. (they all look the same)

Capacity:
  attachable-volumes-aws-ebs:  25
  cpu:                         8
  ephemeral-storage:           32461564Ki
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      65049812Ki
  pods:                        110
Allocatable:
  attachable-volumes-aws-ebs:  25
  cpu:                         8
  ephemeral-storage:           29916577333
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      64947412Ki
  pods:                        110
System Info:
  Machine ID:                 ec223b5ea23ea6bd5b06e8ed0a733d2d
  System UUID:                ec223b5e-a23e-a6bd-5b06-e8ed0a733d2d
  Boot ID:                    798aca5f-d9e1-4c9f-b75d-e16f7ba2d514
  Kernel Version:             5.4.0-1024-aws
  OS Image:                   Ubuntu 20.04.1 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://19.3.11
  Kubelet Version:            v1.18.10
  Kube-Proxy Version:         v1.18.10
Non-terminated Pods:          (5 in total)
  Namespace                   Name                                                        CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                   ----                                                        ------------  ----------  ---------------  -------------  ---
  amazon-cloudwatch           fluentd-cloudwatch-tzsv4                                    100m (1%)     0 (0%)      200Mi (0%)       400Mi (0%)     21d
  default                     prometheus-prometheus-node-exporter-tvmd4                   100m (1%)     0 (0%)      0 (0%)           0 (0%)         21d
  es                          elasticsearch-data-0                                        500m (6%)     1 (12%)     8Gi (12%)        8Gi (12%)      14m
  kube-system                 calico-node-dhxg5                                           100m (1%)     0 (0%)      0 (0%)           0 (0%)         21d
  kube-system                 kube-proxy-ip-10-1-12-115.us-gov-west-1.compute.internal    100m (1%)     0 (0%)      0 (0%)           0 (0%)         21d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                    Requests      Limits
  --------                    --------      ------
  cpu                         900m (11%)    1 (12%)
  memory                      8392Mi (13%)  8592Mi (13%)
  ephemeral-storage           0 (0%)        0 (0%)
  hugepages-1Gi               0 (0%)        0 (0%)
  hugepages-2Mi               0 (0%)        0 (0%)
  attachable-volumes-aws-ebs  0             0

Here is what my cluster looks like

kubectl get pods -n es
NAME                                                 READY   STATUS    RESTARTS   AGE
elasticsearch-client-0                               1/1     Running   0          77m
elasticsearch-client-1                               1/1     Running   0          77m
elasticsearch-data-0                                 1/1     Running   0          77m
elasticsearch-data-1                                 1/1     Running   0          77m
elasticsearch-data-2                                 1/1     Running   0          77m
elasticsearch-data-3                                 0/1     Pending   0          77m
elasticsearch-data-4                                 0/1     Pending   0          77m
elasticsearch-data-5                                 0/1     Pending   0          77m
elasticsearch-data-6                                 0/1     Pending   0          77m
elasticsearch-data-7                                 0/1     Pending   0          77m
elasticsearch-master-0                               2/2     Running   0          77m
elasticsearch-master-1                               2/2     Running   0          77m
prometheus-elasticsearch-exporter-6d6c5d49cf-4w7gc   1/1     Running   0          22h

Here is the events when I describe pod

Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  56s (x5 over 3m35s)  default-scheduler  0/11 nodes are available: 3 Insufficient memory, 3 node(s) didn't match pod affinity/anti-affinity, 3 node(s) didn't satisfy existing pods anti-affinity rules, 5 node(s) didn't match node selector.

Here is my resource limits and requests for the data pods

    Limits:
      cpu:     1
      memory:  8Gi
    Requests:
      cpu:      500m
      memory:   8Gi

Here is what my nodeAffinity looks like

nodeAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
    nodeSelectorTerms:
    - matchExpressions:
      - key: es-data
        operator: In
        values:
        - "true"

And my tolerations

tolerations:
  - key: "es-data"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"

And here is my node tolerance when I describe the node

Taints:             es-data=true:NoSchedule

I tainted it like

kubectl taint nodes <node> es-data=true:NoSchedule

According to my calculations based on my understand (which is probably wrong) my data pods are only asking for 8G of memory from a node which has 64G available, and only one pod requesting 8G of memory is already using it. So it should have theoretically 56G of memory left to other pods requesting to be scheduled to it. And even the memory being used shows me it's only 13% used. Why can't it schedule? How can I troubleshoot? Am I misunderstanding how this should work? What else can I tell you which helps troubleshoot this?

Resolution: Based on Hakob's comments, the issue is that I had nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution set, which is a hard requirement directing the scheduler to only schedule one to each node. What I needed to do in order to schedule more than one to each node was change it to nodeSelector. If you are reaching this, please note Hakob's recommendation for why it is not suggested best practice to do this. I agree with that advice. Though in my case it is a requirement coming from client and did not have the option even after have discussion why they should not be doing this. So please take that into consideration when applying this change.

3
  • Probably the anti-affinity rule trigger the memory rule along it. I'd first work on the affinity problem first. – Tensibai Mar 6 at 10:20
  • Could you elaborate please. The affinity should be ok, since the first three are getting scheduled to the appropriate memory optimized tainted nodes just fine. Shouldn't they? And they never get scheduled where they are not suppose to. No other pods are scheduled to these memory optimized nodes. So everything seems just as I expected, until more than 1 of these pods try to get scheduled to one of the memory optimized. – protobyte Mar 6 at 16:56
  • Roughly what is in the answer: you ask for your pods to NOT be on the same node, so obviously it can only start as much pods as you have nodes. – Tensibai Mar 7 at 23:35
2

3 Insufficient memory, 3 node(s) didn't match pod affinity/anti-affinity, 3 node(s) didn't satisfy existing pods anti-affinity rules

This means that ES trying to find a different node to deploy separately all the pods of ES. But the cause of your node count is not enough to run one pod on each node, the other pods remain pending state.

For more information read here

So from here, you have a 2 choice Neo))

  1. Add more nodes until all pods will have been scheduled on the different nodes
  2. Minimize your ES sts pod count to 3
8
  • Is there not a way to have more than one es data pod running on each of the r5 instances? I specifically got the r5.2xl's so that I could run up to roughly 8 es data pods. Other general usage nodes that I am running can run more than one of any kind of pod if there is enough node resources. What makes these tainted nodes only able to run one es data pod at a time. It shouldn't be the nodes resources from what I'm seeing. – protobyte Mar 7 at 18:21
  • You can run more than one pod but it's highly not recommended and doesn't lie under the best practices of DevOps, mainly due to the high availability and fault tolerance. – Hakob Mar 7 at 18:46
  • Ok. That is fair enough. It's mostly trying to test right now to make sure that we can autoscale the pods up before scaling out the nodes. Also that is why there are three different r5 nodes also running es data pods. So one could go down and I'd still have 2 more. So aside from not recommended, and k8s doing this on other nodes/pods by default. Can you explain what it is in my setup I've described which is preventing it from scheduling more than one of the same pod per node? – protobyte Mar 7 at 19:02
  • devops.stackexchange.com/a/13497/10769 do you read the link above? – Hakob Mar 7 at 19:06
  • If I'm understanding it correctly. I need to change the nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution to nodeSelector? – protobyte Mar 7 at 19:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.