14 votes
Accepted

Prometheus alert CPUThrottlingHigh raised but monitoring does not show it

The CPUThrottlingHigh is an alert created by the kubernetes-mixin project. There is an open issue (#108) to discuss this alert. I suggest that you read all the comments on this issue to better ...
user avatar
13 votes
Accepted

Why is Prometheus not a good choice for data with high cardinality?

As far as I know, Prometheus doesn't mind high-cardinality data. What Prometheus doesn't like is high-cardinality labels. Let's start with Prometheus official documentation, it gives a good high-...
user avatar
  • 1,945
8 votes
Accepted

How fast does Prometheus data grow?

Prometheus use a timeseries databases with vacuum, the documentation gives some maths to plan your disk consumption: On average, Prometheus uses only around 1-2 bytes per sample. Thus, to plan ...
user avatar
  • 11.1k
7 votes
Accepted

How to calculate burn rate for SLOs?

Yes, you are interpreting that correctly. When you express your SLO as a percentage, you should express your error rate as a percentage too. So, in you example your error rate of 1 is 100%. That would ...
user avatar
6 votes

How to calculate disk space required by Prometheus v2.2?

To calculate disk space required by Prometheus v2.20 in bytes, use the query: retention_time_seconds * rate(prometheus_tsdb_head_samples_appended_total[2h]) * (rate(...
user avatar
  • 161
5 votes

Keeping graphs of terminated Kubernetes pods in Prometheus/Grafana

The metrics for deleted pods still exists on the Prometheus Time Series Database. The problem is that the Grafana dashboards imported from kubernetes-mixin create the pod list when loading the ...
user avatar
5 votes

How do I troubleshoot missing data in my Prometheus database?

I think your can do some kind of alerting on a metric rate with something like this: ALERT DropInMetricsFromExporter IF rate(<metric_name>[1m]) == 0 FOR 3m ANNOTATIONS { summary = "...
user avatar
  • 11.1k
3 votes

What is the Prometheus and Grafana ideal setup?

I don't think there's an ideal solution at all. However, there may be a preferred one, depending on circumstances. There're several factors which I think one has to consider: How cheap/expensinve a ...
user avatar
3 votes
Accepted

What is Unit testing in the context of YML configuration Alerting rules?

The important part of the blog post you link to isn't testing the syntax of the YAML file; it's testing the semantics of the Prometheus configuration. The core configuration in that post includes a ...
user avatar
3 votes
Accepted

How to calculate disk space required by Prometheus v2.2?

I think your math may be off. If you use the ~18600 number you found I get a very different result from 68TiB: 2,592,000 (seconds) * 18600 (samples/second) * 1.3 (bytes/sample) = 62,674,560,000 (...
user avatar
3 votes

How to get hostname from node-exporter in Docker Swarm?

You just have to use the hostname on your docker compose file like this: hostname: whatever-{{.Node.Hostname}} Note that you have to prepend something to the hostname to avoid hostnames duplicates.
user avatar
  • 31
3 votes
Accepted

Local Prometheus and Grafana Installations to Monitor a Kubernetes Cluster

It's possible but after doing many replatforming projects for clients I can't say I would advise it. Have you seen kube-prometheus? It's a one-shot install that will get you 90% of where you need to ...
user avatar
  • 759
3 votes
Accepted

Prometheus Alertamanger - how to silence all alerts for a given period during a maintenance?

What about Name = "prometheus", Value = ".*". That is something that worked for me. An alternative can be Name = "severity", Value = ".*" also.
user avatar
3 votes

How prometheus scrapes individual pods actuator endpoints when they are exposed via a single service in kubernetes?

No changes need to be made to your service deployment. As for Prometheus, if you are deploying a Prometheus Operator, A simple ServiceMonitor will work for you. ServiceMonitors select the services to ...
user avatar
2 votes

How to find interdependencies between pods in a Kubernetes cluster?

Have a look to solutions implementing a service mesh. For example ISTIO, Consul, Ambasador + Envoy...
user avatar
  • 161
2 votes

How to find interdependencies between pods in a Kubernetes cluster?

Weave Scope is an observability tool that provides a dashboard that shows all network connections between pods. https://github.com/weaveworks/scope
user avatar
  • 1,345
2 votes

Prometheus alert CPUThrottlingHigh raised but monitoring does not show it

To piggy back off of @eduardo-baitello answer, A third option is to increase the CPUThrottlingPercent config here
user avatar
2 votes
Accepted

Which libraries generate observability metrics for applications in the cluster to prometheus?

For Application specific calls you need to use a Prometheus client and create the metrics yourself. The Prometheus client will allow you to create a /metrics endpoint where you can output the current ...
user avatar
  • 1,269
2 votes

Jenkins plugin recomendation

Have a look at the Summary Display plugin. Some more details about it (= quote from the linked page): ... allows an easy and fully customized build report display. Features Allow a rich summary ...
user avatar
  • 1,361
2 votes
Accepted

Back-filling prometheus (and related system) metrics?

Answering my own question (after a few days of research). I have yet to test this but https://github.com/Cleafy/promqueen seems to be designed for this purpose: PromQueen made possible to record ...
user avatar
2 votes

Using Prometheus to monitor Spring Boot Applications in Kubernetes Cluster

The best way to deploy prometheus to kubernetes is with the helm chart: https://github.com/helm/charts/tree/master/stable/prometheus If you haven't used helm yet, its a very simple install to your ...
user avatar
2 votes

Prometheus Alertamanger - how to silence all alerts for a given period during a maintenance?

With alertmanager 0.22 (perhaps even earlier?) the solution using regex is a bit less obvious as there is no longer any checkbox to tick for using a regex. Instead, you must use Name="Value" ...
user avatar
  • 121
2 votes
Accepted

How to filter label_values in Loki?

After researching this, i've found an open issue on grafana's github. At the time of writing its still in the backlog.
user avatar
2 votes
Accepted

Enable ingress-nginx Prometheus monitoring without Helm

Your ServiceMonitor seems to be correct with the exception that you have defined no path under spec.endpoints. You also need to define the port named prometheus under the ingress-nginx service or ...
user avatar
2 votes
Accepted

Scraping different Prometheus metrics at different rates

You can define a different scrape_interval in the scrape config of two different jobs: a job for slow polling and one for fast polling. If your metrics are on different endpoints you can simply adjust ...
user avatar
1 vote
Accepted

Reuse Alertmanger Slack Integration into Jiralert template

In the end solved it myself, I had to just define jira.summary and description respectively and copy/paste the contents from the slack integration in the given fields. {{ define "jira.summary" }}] ...
user avatar
1 vote

Pometheus-operator - modify Grafana mixin dashboard

You should use https://github.com/helm/charts/tree/master/stable/prometheus-operator helm chart , also you can write Grafana dashboard script that would create dashboard dynamically , can be update as ...
user avatar
1 vote
Accepted

Prometheus - join series based on the key part of a label

label_values(kube_node_role, role) sum(kube_pod_container_resource_requests_cpu_cores * on (node) group_left (role) kube_node_role{role="$node_role"}) kube_node_role out of kube-state-metrics can ...
user avatar
1 vote

What is the Prometheus and Grafana ideal setup?

You covered the obvious security issue, use HTTPS for all your connections. While I won't try to tell you how to implement your environment, I will share with you the design in which I am rolling ...
user avatar
1 vote

Expose metrics for Kotlin using JMX_Exporter to prometheus

After trying many things, I came to know that I needed to expose the port of my application's container in order to let Prometheus or other deployments to know. After exposing the port, I could see my ...
user avatar
  • 121

Only top scored, non community-wiki answers of a minimum length are eligible