15
votes
Accepted
Prometheus alert CPUThrottlingHigh raised but monitoring does not show it
The CPUThrottlingHigh is an alert created by the kubernetes-mixin project. There is an open issue (#108) to discuss this alert. I suggest that you read all the comments on this issue to better ...
- 961
13
votes
Accepted
Why is Prometheus not a good choice for data with high cardinality?
As far as I know, Prometheus doesn't mind high-cardinality data. What Prometheus doesn't like is high-cardinality labels.
Let's start with Prometheus official documentation, it gives a good high-...
- 1,965
8
votes
How to calculate disk space required by Prometheus v2.2?
To calculate disk space required by Prometheus v2.20 in bytes, use the query:
retention_time_seconds *
rate(prometheus_tsdb_head_samples_appended_total[2h]) *
(rate(...
- 181
8
votes
Accepted
How to calculate burn rate for SLOs?
Yes, you are interpreting that correctly.
When you express your SLO as a percentage, you should express your error rate as a percentage too. So, in you example your error rate of 1 is 100%. That would ...
- 106
8
votes
Accepted
How fast does Prometheus data grow?
Prometheus use a timeseries databases with vacuum, the documentation gives some maths to plan your disk consumption:
On average, Prometheus uses only around 1-2 bytes per sample. Thus, to
plan ...
- 11.4k
6
votes
Prometheus Alertamanger - how to silence all alerts for a given period during a maintenance?
With alertmanager 0.22 (perhaps even earlier?) the solution using regex is a bit less obvious as there is no longer any checkbox to tick for using a regex.
Instead, you must use Name="Value" ...
- 161
5
votes
Keeping graphs of terminated Kubernetes pods in Prometheus/Grafana
The metrics for deleted pods still exists on the Prometheus Time Series Database. The problem is that the Grafana dashboards imported from kubernetes-mixin create the pod list when loading the ...
- 961
5
votes
How do I troubleshoot missing data in my Prometheus database?
I think your can do some kind of alerting on a metric rate with something like this:
ALERT DropInMetricsFromExporter
IF rate(<metric_name>[1m]) == 0
FOR 3m
ANNOTATIONS {
summary = "...
- 11.4k
5
votes
Accepted
Prometheus Alertamanger - how to silence all alerts for a given period during a maintenance?
If it's the only selector alertmanager (0.24) doesn't allow it to match an empty string.
So use prometheus=~".+" or severity=~".+" instead of prometheus=~".*", otherwise ...
- 66
4
votes
Accepted
How to calculate disk space required by Prometheus v2.2?
I think your math may be off. If you use the ~18600 number you found I get a very different result from 68TiB:
2,592,000 (seconds) * 18600 (samples/second) * 1.3 (bytes/sample) = 62,674,560,000 (...
- 2,762
4
votes
Prometheus Alertamanger - how to silence all alerts for a given period during a maintenance?
What about Name = "prometheus", Value = ".*". That is something that worked for me.
An alternative can be Name = "severity", Value = ".*" also.
- 246
3
votes
Accepted
Back-filling prometheus (and related system) metrics?
Answering my own question (after a few days of research).
I have yet to test this but https://github.com/Cleafy/promqueen seems to be designed for this purpose:
PromQueen made possible to record ...
- 1,540
3
votes
What is the Prometheus and Grafana ideal setup?
I don't think there's an ideal solution at all. However, there may be a preferred one, depending on circumstances.
There're several factors which I think one has to consider:
How cheap/expensinve a ...
3
votes
Accepted
What is Unit testing in the context of YML configuration Alerting rules?
The important part of the blog post you link to isn't testing the syntax of the YAML file; it's testing the semantics of the Prometheus configuration.
The core configuration in that post includes a ...
- 521
3
votes
How to get hostname from node-exporter in Docker Swarm?
You just have to use the hostname on your docker compose file like this:
hostname: whatever-{{.Node.Hostname}}
Note that you have to prepend something to the hostname to avoid hostnames duplicates.
- 31
3
votes
Accepted
Local Prometheus and Grafana Installations to Monitor a Kubernetes Cluster
It's possible but after doing many replatforming projects for clients I can't say I would advise it. Have you seen kube-prometheus? It's a one-shot install that will get you 90% of where you need to ...
- 1,054
3
votes
How prometheus scrapes individual pods actuator endpoints when they are exposed via a single service in kubernetes?
No changes need to be made to your service deployment. As for Prometheus, if you are deploying a Prometheus Operator, A simple ServiceMonitor will work for you. ServiceMonitors select the services to ...
- 421
3
votes
Accepted
Does Prometheus log scrape-duration?
Yes, you can use the scrape_duration_seconds metric.
Then you can run queries in the Prometheus query box like:
scrape_duration_seconds{instance="localhost:9100", job="node"}[5m]
...
- 171
2
votes
How to find interdependencies between pods in a Kubernetes cluster?
Have a look to solutions implementing a service mesh.
For example ISTIO, Consul, Ambasador + Envoy...
- 161
2
votes
How to find interdependencies between pods in a Kubernetes cluster?
Weave Scope is an observability tool that provides a dashboard that shows all network connections between pods.
https://github.com/weaveworks/scope
- 1,405
2
votes
Prometheus alert CPUThrottlingHigh raised but monitoring does not show it
To piggy back off of @eduardo-baitello answer, A third option is to increase the CPUThrottlingPercent config here
2
votes
Accepted
Which libraries generate observability metrics for applications in the cluster to prometheus?
For Application specific calls you need to use a Prometheus client and create the metrics yourself. The Prometheus client will allow you to create a /metrics endpoint where you can output the current ...
- 1,279
2
votes
Jenkins plugin recomendation
Have a look at the Summary Display plugin. Some more details about it (= quote from the linked page):
... allows an easy and fully customized build report display.
Features
Allow a rich summary ...
- 1,556
2
votes
Using Prometheus to monitor Spring Boot Applications in Kubernetes Cluster
The best way to deploy prometheus to kubernetes is with the helm chart:
https://github.com/helm/charts/tree/master/stable/prometheus
If you haven't used helm yet, its a very simple install to your ...
- 1,540
2
votes
Accepted
Prometheus - join series based on the key part of a label
label_values(kube_node_role, role)
sum(kube_pod_container_resource_requests_cpu_cores
* on (node) group_left (role) kube_node_role{role="$node_role"})
kube_node_role out of kube-state-metrics can ...
- 101
2
votes
Accepted
How to filter label_values in Loki?
After researching this, i've found an open issue on grafana's github. At the time of writing its still in the backlog.
- 151
2
votes
Where to install Prometheus for Kubernetes monitoring
This is more of an opinion based question but let me try to answer it.
installing Prometheus itself in the very same k8s cluster (as another running set of pods) is a normal thing.Am I right or just ...
- 421
2
votes
Accepted
Enable ingress-nginx Prometheus monitoring without Helm
Your ServiceMonitor seems to be correct with the exception that you have defined no path under spec.endpoints.
You also need to define the port named prometheus under the ingress-nginx service or ...
- 421
2
votes
Accepted
Scraping different Prometheus metrics at different rates
You can define a different scrape_interval in the scrape config of two different jobs: a job for slow polling and one for fast polling.
If your metrics are on different endpoints you can simply adjust ...
- 136
2
votes
Accepted
What is the difference between Fluent Bit and Prometheus?
This is a little off. You are comparing a "Time Series Database" to a "Log Shipper".
How Prometheus Works
Any service/entity prometheus wants to run monitor has to provide an &...
- 1,540
Only top scored, non community-wiki answers of a minimum length are eligible