2

I love using Prometheus/Grafana to monitor and tune resource usage on my Kubernetes clusters. When pods are terminated, however, the graphs are disposed of it seems.

I am looking for a solution to this problem.

For informed decisions on how to tune resource requests I need history to be kept.

Does anybody know how to keep graphs for deleted Kubernetes pods in Prometheus/Grafana.

I am currently using this deployment: https://itnext.io/kubernetes-monitoring-with-prometheus-in-15-minutes-8e54d1de2e13

Improve this question

1 Answer 1

Reset to default
5

The metrics for deleted pods still exists on the Prometheus Time Series Database. The problem is that the Grafana dashboards imported from kubernetes-mixin create the pod list when loading the dashboards.

To change this in Grafana, open the Variables menu, then change $pod variable to refresh On Time Range Change:

enter image description here

Go back to the panels and change the time range (e.g., to "This Month"). The deleted pods for that period should be listed in the dropdown menu.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.