micrometer exposing actuator metrics to set request/limit to pods in K8svs metrics-server vs kube-state-metrics -> K8s Mixin from kube-promethteus-stack Grafana dashboad It's really blurry and frustrating to me to understand why there is such big difference between values from the 3 in the title and how should one utilize K8s Mixin to set proper request/limits and if that is expected at al. I was hoping I can just see same data that I see when I type kubectl top podname --containers to what I see when I open K8s -> ComputeResources -> Pods dashboard in Grafana. But not only the values differ by more than a double, but also reported values from actuator differ from both. When exposing spring data with micrometer the sum of jvm_memory_used_bytes is corresponding more to what I get from metrics-server (0.37.0) rather then what I see on Grafana from the mixin dashboards, but it is still far off. I am using K8s: 1.14.3 on Ubuntu 18.04 LTS managed by kubespray. kube-prometheus-stack 9.4.4 installed with helm 2.14.3. Spring boot 2.0 with Micrometer. I saw the explanation on metrics-server git that this is the value that kubelet use for OOMKill, but again this is not helpful at all as what should I do with the dashboard? What is the the way to handle this?
-
I found out the double values are result of something in the provided querry in Grafana: When I run the following instant vector container_memory_working_set_bytes{cluster="", namespace="qa2", pod="service-x-684f5565b7-wbgm6", container!="POD", container!="", image!=""} why would I get 4 results instead of 2 (service and istio-proxy), what is the logical explanation behind this, I get 2 time the service and 2 time the proxy with same values, but different id in the metrics endpoint is that a bug in kube-prometheus-stack helm chart or perhaps something on operator level?– anVzdGFub3RoZXJodW1hbgOct 15, 2020 at 14:42
1 Answer
Based on what I see so far, I have found the root cause, renamed kubelet service from old chart to new that can get targeted by serviceMonitors. So for me the best solution would be grafana kube-state-metrics + comparing what I see in the jvm dashboard