How to monitor ELK using metricbeat on kubernetes (v8 compatible and simple solution)

Question

For probably more than a year there's been a deprecation warning on xpack monitoring in elasticsearch, logstash, kibana etc. saying the built-in monitoring is going to disappear.

Since I don't like running deprecated setups I figured it was time to fix it now, should be easy, right? Not really, what is 2 lines of config in ELK 7.x instead requires deploying an extra metricbeat instance. Personally I really don't like this approach since it's a hassle and seems like a waste of resources to deploy another container just to send off some monitoring metrics. But elastic has chosen the path so we can just follow.

I read a bit in some official channels and there seems to be some official solutions in the works that require custom kubernetes ELK CRDs etc. This is not what I want, I just want a simple solution to deploy anywhere as a sidecar.

Answer 1

I propose the following solution although it will require 1 extra metricbeat running per monitored entity, it's simple, not optimized. The approach will also work for other non-kubernetes contexts like docker-compose with a little bit of tweaking.

• Deploy sidecar (extra container per pod) to monitor something (in this example, it's logstash but doing same for elasticsearch or kibana should be trivial, just replacing the monitoring module and possbibly metrics port.
• Set xpack.monitoring.enabled: false in logstash.yml (or elastic/kibana)
• Have metricbeat fetch metrics via localhost on port 9600
• Configure metricbeat with mounted configmaps defined in yaml
• Example assumes no TLS/RBAC to elasticsearch but if you have tls modifying the below should hopefully be straight forward

# Configmaps for metricbeat conf files
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-modules
  namespace: logstash
data:
  logstash-xpack.yml: |-
    - module: logstash
      xpack.enabled: true # This is what turns on xpack monitoring metric export
      period: 10s
      hosts: ["localhost:9600"] # Connect to logstash metrics api using localhost inside pod
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-config
  namespace: logstash
data:
  metricbeat.yml: |-
    metricbeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: true
      reload.period: 10s
    metricbeat.modules: # Disable system metrics
    - module: system
      enabled: false

    output.elasticsearch:
      hosts: ["http://elasticsearch:9200"] # You probably need to modify elasticsearch URL 
    setup.ilm.enabled: true

---
...
Here goes the rest of your yaml for logstash/elasticsearch/kibana
And then the sidecar container:
... 
      containers:
      - name: metricbeat-sidecar
        image: docker.elastic.co/beats/metricbeat:7.14.0
        volumeMounts:
        - name: config
          mountPath: /usr/share/metricbeat/metricbeat.yml
          readOnly: true
          subPath: metricbeat.yml
        - name: modules
          mountPath: /usr/share/metricbeat/modules.d
          readOnly: true
...
# and then finally in the volumes section:
...
      volumes:
      - name: config
        configMap:
          defaultMode: 0640
          name: metricbeat-config
      - name: modules
        configMap:
          defaultMode: 0640
          name: metricbeat-modules

Main sources:

• https://stackoverflow.com/questions/67145400/creating-sidecar-metricbeat-with-aws-eks-fargate
• https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html

Update: Similarly for kibana:

# Configmaps for metricbeat conf files
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kibana-metricbeat-modules
  namespace: elk
data:
  xpack.yml: |-
    - module: kibana
      xpack.enabled: true # This is what turns on reporting
      period: 10s
      hosts: ["localhost:5601"]
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kibana-metricbeat-config
  namespace: elk
data:
  metricbeat.yml: |-
    metricbeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: true
      reload.period: 10s
    metricbeat.modules: # Disable system metrics
    - module: system
      enabled: false

    output.elasticsearch:
      hosts: ["http://elasticsearch:9200"]
    setup.ilm.enabled: true
...
container: 
  - name: metricbeat-sidecar
      image: docker.elastic.co/beats/metricbeat:7.14.0
      volumeMounts:
        - name: config
          mountPath: /usr/share/metricbeat/metricbeat.yml
          readOnly: true
          subPath: metricbeat.yml
        - name: modules
          mountPath: /usr/share/metricbeat/modules.d
          readOnly: true
...
      volumes:
    - name: config
      configMap:
        defaultMode: 0640
        name: kibana-metricbeat-config
    - name: modules
      configMap:
        defaultMode: 0640
        name: kibana-metricbeat-modules