Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more
15 votes
Accepted

Prometheus alert CPUThrottlingHigh raised but monitoring does not show it

The CPUThrottlingHigh is an alert created by the kubernetes-mixin project. There is an open issue (#108) to discuss this alert. I suggest that you read all the comments on this issue to better ...
Eduardo Baitello's user avatar
13 votes
Accepted

Why is Prometheus not a good choice for data with high cardinality?

As far as I know, Prometheus doesn't mind high-cardinality data. What Prometheus doesn't like is high-cardinality labels. Let's start with Prometheus official documentation, it gives a good high-...
Alexandre's user avatar
  • 1,965
8 votes

How to calculate disk space required by Prometheus v2.2?

To calculate disk space required by Prometheus v2.20 in bytes, use the query: retention_time_seconds * rate(prometheus_tsdb_head_samples_appended_total[2h]) * (rate(...
Brandon's user avatar
  • 181
8 votes
Accepted

How to calculate burn rate for SLOs?

Yes, you are interpreting that correctly. When you express your SLO as a percentage, you should express your error rate as a percentage too. So, in you example your error rate of 1 is 100%. That would ...
Marty Pauley's user avatar
8 votes
Accepted

How fast does Prometheus data grow?

Prometheus use a timeseries databases with vacuum, the documentation gives some maths to plan your disk consumption: On average, Prometheus uses only around 1-2 bytes per sample. Thus, to plan ...
Tensibai's user avatar
  • 11.4k
6 votes

Prometheus Alertamanger - how to silence all alerts for a given period during a maintenance?

With alertmanager 0.22 (perhaps even earlier?) the solution using regex is a bit less obvious as there is no longer any checkbox to tick for using a regex. Instead, you must use Name="Value" ...
wjentner's user avatar
  • 161
5 votes

Keeping graphs of terminated Kubernetes pods in Prometheus/Grafana

The metrics for deleted pods still exists on the Prometheus Time Series Database. The problem is that the Grafana dashboards imported from kubernetes-mixin create the pod list when loading the ...
Eduardo Baitello's user avatar
5 votes

How do I troubleshoot missing data in my Prometheus database?

I think your can do some kind of alerting on a metric rate with something like this: ALERT DropInMetricsFromExporter IF rate(<metric_name>[1m]) == 0 FOR 3m ANNOTATIONS { summary = "...
Tensibai's user avatar
  • 11.4k
4 votes
Accepted

How to calculate disk space required by Prometheus v2.2?

I think your math may be off. If you use the ~18600 number you found I get a very different result from 68TiB: 2,592,000 (seconds) * 18600 (samples/second) * 1.3 (bytes/sample) = 62,674,560,000 (...
Wesley Rolnick's user avatar
4 votes

Prometheus Alertamanger - how to silence all alerts for a given period during a maintenance?

What about Name = "prometheus", Value = ".*". That is something that worked for me. An alternative can be Name = "severity", Value = ".*" also.
Mikhail Advani's user avatar
3 votes
Accepted

Back-filling prometheus (and related system) metrics?

Answering my own question (after a few days of research). I have yet to test this but https://github.com/Cleafy/promqueen seems to be designed for this purpose: PromQueen made possible to record ...
John Humphreys's user avatar
3 votes

What is the Prometheus and Grafana ideal setup?

I don't think there's an ideal solution at all. However, there may be a preferred one, depending on circumstances. There're several factors which I think one has to consider: How cheap/expensinve a ...
Alexander Pushkarev's user avatar
3 votes
Accepted

What is Unit testing in the context of YML configuration Alerting rules?

The important part of the blog post you link to isn't testing the syntax of the YAML file; it's testing the semantics of the Prometheus configuration. The core configuration in that post includes a ...
David Maze's user avatar
3 votes

How to get hostname from node-exporter in Docker Swarm?

You just have to use the hostname on your docker compose file like this: hostname: whatever-{{.Node.Hostname}} Note that you have to prepend something to the hostname to avoid hostnames duplicates.
gespert's user avatar
  • 31
3 votes
Accepted

Local Prometheus and Grafana Installations to Monitor a Kubernetes Cluster

It's possible but after doing many replatforming projects for clients I can't say I would advise it. Have you seen kube-prometheus? It's a one-shot install that will get you 90% of where you need to ...
Levi's user avatar
  • 1,054
3 votes
Accepted

Prometheus Alertamanger - how to silence all alerts for a given period during a maintenance?

If it's the only selector alertmanager (0.24) doesn't allow it to match an empty string. So use prometheus=~".+" or severity=~".+" instead of prometheus=~".*", otherwise ...
pa7ac's user avatar
  • 46
3 votes

How prometheus scrapes individual pods actuator endpoints when they are exposed via a single service in kubernetes?

No changes need to be made to your service deployment. As for Prometheus, if you are deploying a Prometheus Operator, A simple ServiceMonitor will work for you. ServiceMonitors select the services to ...
rohatgisanat's user avatar
3 votes
Accepted

Does Prometheus log scrape-duration?

Yes, you can use the scrape_duration_seconds metric. Then you can run queries in the Prometheus query box like: scrape_duration_seconds{instance="localhost:9100", job="node"}[5m] ...
Kate's user avatar
  • 146
2 votes

How to find interdependencies between pods in a Kubernetes cluster?

Have a look to solutions implementing a service mesh. For example ISTIO, Consul, Ambasador + Envoy...
DevLounge's user avatar
  • 161
2 votes

How to find interdependencies between pods in a Kubernetes cluster?

Weave Scope is an observability tool that provides a dashboard that shows all network connections between pods. https://github.com/weaveworks/scope
user2640621's user avatar
  • 1,395
2 votes

Prometheus alert CPUThrottlingHigh raised but monitoring does not show it

To piggy back off of @eduardo-baitello answer, A third option is to increase the CPUThrottlingPercent config here
Saleh Rastani's user avatar
2 votes
Accepted

Which libraries generate observability metrics for applications in the cluster to prometheus?

For Application specific calls you need to use a Prometheus client and create the metrics yourself. The Prometheus client will allow you to create a /metrics endpoint where you can output the current ...
avi's user avatar
  • 1,279
2 votes

Jenkins plugin recomendation

Have a look at the Summary Display plugin. Some more details about it (= quote from the linked page): ... allows an easy and fully customized build report display. Features Allow a rich summary ...
Subhash's user avatar
  • 1,536
2 votes

Using Prometheus to monitor Spring Boot Applications in Kubernetes Cluster

The best way to deploy prometheus to kubernetes is with the helm chart: https://github.com/helm/charts/tree/master/stable/prometheus If you haven't used helm yet, its a very simple install to your ...
John Humphreys's user avatar
2 votes
Accepted

Prometheus - join series based on the key part of a label

label_values(kube_node_role, role) sum(kube_pod_container_resource_requests_cpu_cores * on (node) group_left (role) kube_node_role{role="$node_role"}) kube_node_role out of kube-state-metrics can ...
Brando___'s user avatar
  • 101
2 votes
Accepted

How to filter label_values in Loki?

After researching this, i've found an open issue on grafana's github. At the time of writing its still in the backlog.
user2324712's user avatar
2 votes
Accepted

Enable ingress-nginx Prometheus monitoring without Helm

Your ServiceMonitor seems to be correct with the exception that you have defined no path under spec.endpoints. You also need to define the port named prometheus under the ingress-nginx service or ...
rohatgisanat's user avatar
2 votes
Accepted

Scraping different Prometheus metrics at different rates

You can define a different scrape_interval in the scrape config of two different jobs: a job for slow polling and one for fast polling. If your metrics are on different endpoints you can simply adjust ...
Michael Doubez's user avatar
2 votes

List all active promtail targets

Sounds like you want ${PROMTAIL_HTTP_ENDPOINT}/targets. See the server code
Bruce Becker's user avatar
  • 3,593
2 votes
Accepted

Measuring image pull latency in k8s cluster

For Kubernetes >=1.30 Kubelet now exports a histogram, image_pull_duration_seconds, for exactly this case. See kubernetes/kubernetes#121719 for details.
Elad Avikzer's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible