0

I'm specifying an Elasticsearch cluster with the elasticsearch.k8s.elastic.co/v1/Elasticsearch from Elastic Cloud on Kubernetes 2.12 CRD like

spec:
  nodeSets:
    - config:
        node.store.allow_mmap: false
      count: 3
      name: default
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 75Gi

which at some point leads to a situation where memory is no longer sufficient and the Cluster goes into a circuit-breaker exception state where all inter-cluster communication fails with code 429 and the infamous error "Data too large" indicating insufficient memory.

It's time to increase the memory by specifying it in podTemplate like (patch)

spec:
  nodeSets:
    - name: default
      podTemplate:
        spec:
          containers:
            - name: elasticsearch
              resources:
                requests:
                  cpu: 2
                  memory: 4Gi

You can also specify Xmx in env.ES_JAVA_OPTS, but that's irrelevant to my problem that the cluster events now show that elasticsearch-controller cannot update the underlying Statefulset due to Unsupported Elasticsearch distribution: unable to verify Elasticsearch license: elasticsearch client failed for https://logging-cluster-es-internal-http.logging.svc:9200/_license: 429 Too Many Requests: {Status:429 Error:{CausedBy:{Reason: Type:} Reason:[parent] Data too large, data for [cluster:monitor/xpack/license/get] would be [1684859172/1.5gb], which is larger than the limit of [1653604352/1.5gb], real usage: [1684859104/1.5gb], new bytes reserved: [68/68b], usages [inflight_requests=68/68b, model_inference=0/0b, eql_sequence=0/0b, fielddata=263/263b, request=0/0b] Type:circuit_breaking_exception StackTrace: RootCause:[{Reason:[parent] Data too large, data for [cluster:monitor/xpack/license/get] would be [1684859172/1.5gb], which is larger than the limit of [1653604352/1.5gb], real usage: [1684859104/1.5gb], new bytes reserved: [68/68b], usages [inflight_requests=68/68b, model_inference=0/0b, eql_sequence=0/0b, fielddata=263/263b, request=0/0b] Type:circuit_breaking_exception}]}} which effectively means that in order to increase memory you need more memory.

I consider this a configuration deadlock. How to get out it?

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.