0

micrometer exposing actuator metrics to set request/limit to pods in K8svs metrics-server vs kube-state-metrics -> K8s Mixin from kube-promethteus-stack Grafana dashboad It's really blurry and frustrating to me to understand why there is such big difference between values from the 3 in the title and how should one utilize K8s Mixin to set proper request/limits and if that is expected at al. I was hoping I can just see same data that I see when I type kubectl top podname --containers to what I see when I open K8s -> ComputeResources -> Pods dashboard in Grafana. But not only the values differ by more than a double, but also reported values from actuator differ from both. When exposing spring data with micrometer the sum of jvm_memory_used_bytes is corresponding more to what I get from metrics-server (0.37.0) rather then what I see on Grafana from the mixin dashboards, but it is still far off. I am using K8s: 1.14.3 on Ubuntu 18.04 LTS managed by kubespray. kube-prometheus-stack 9.4.4 installed with helm 2.14.3. Spring boot 2.0 with Micrometer. I saw the explanation on metrics-server git that this is the value that kubelet use for OOMKill, but again this is not helpful at all as what should I do with the dashboard? What is the the way to handle this?

enter image description here

enter image description here

enter image description here

Improve this question
1
  • I found out the double values are result of something in the provided querry in Grafana: When I run the following instant vector container_memory_working_set_bytes{cluster="", namespace="qa2", pod="service-x-684f5565b7-wbgm6", container!="POD", container!="", image!=""} why would I get 4 results instead of 2 (service and istio-proxy), what is the logical explanation behind this, I get 2 time the service and 2 time the proxy with same values, but different id in the metrics endpoint is that a bug in kube-prometheus-stack helm chart or perhaps something on operator level?
    – anVzdGFub3RoZXJodW1hbg
    Oct 15, 2020 at 14:42

1 Answer 1

Reset to default
0

Based on what I see so far, I have found the root cause, renamed kubelet service from old chart to new that can get targeted by serviceMonitors. So for me the best solution would be grafana kube-state-metrics + comparing what I see in the jvm dashboard

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.