Questions tagged [prometheus]

For questions about Prometheus, an open-source systems monitoring and alerting toolkit.

Filter by
Sorted by
Tagged with
0
votes
1answer
11 views

How to filter label_values in Loki?

I want to create a query variable in grafana using Loki that returns all the possible label values of the label baz such that they only include label values that exist where the label foo equals "...
0
votes
1answer
15 views

Node-centric network metrics are very different from the sum of metrics of the Pods running on that node

Context I am trying to chart the network bandwidth usage of a node in 2 different manners: By looking at global metrics for that node By summing up the corresponding metric for each Pod To achieve ...
0
votes
0answers
14 views

Unable to get traefik running with prometheus and grafana on docker swarm

I have asked this question at official Traefik Community Forum and I have not received any replies there yet so I am hoping I have better luck here :) I am trying to run a traefik-grafana-prometheus ...
0
votes
0answers
13 views

Grafana lost access to previous data of prometheus

I'm gathering bandwidth data of some servers using prometheus federation and grafana for monitoring. Last week I noticed that something weird happened to my grafana dashboard, seems garafan can't have ...
0
votes
0answers
12 views

What is data resolution? High-resolution and Low-resolution data?

What is data resolution? High-resolution and Low-resolution data? I can imagine what it could be based on image resolution, but I am not quite sure. In connection with TSDB (e.g. Prometheus) I have ...
0
votes
0answers
11 views

What is blackbox_exporter probe_http_uncompressed_body_length exactly?

What is the metric probe_http_uncompressed_body_length exactly? How to reproduce this metric with curl? Why is it 0 at me? Not all my targets are 0, but many targets that return a small JSON document ...
0
votes
1answer
11 views

micrometer exposing actuator metrics vs kube-state-metrics vs metrics-server to set pod request/limits

micrometer exposing actuator metrics to set request/limit to pods in K8svs metrics-server vs kube-state-metrics -> K8s Mixin from kube-promethteus-stack Grafana dashboad It's really blurry and ...
0
votes
2answers
53 views

Is it possible/fine to run Prometheus, Loki, Grafana outside of Kubernetes?

In some project there are scaling and orchestration implemented using technologies of a local cloud provider, with no Docker & Kubernetes. But the project has poor logging and monitoring, I'd like ...
0
votes
1answer
9 views

Grafana Status Panel Plugin - genereate status box for each entry from a Prometheus querry

I am trying to create a dashboad out of process_uptime_seconds{namespace="$namespace"} querry from Spring Actuator. and Prometheus as datasource I am trying to build a single stat box menus ...
0
votes
0answers
18 views

Prometheus two source labels relabeling

Is it possible to change __address__ port if prometheus job name is equal to node-exporter? Something like in example below but its not working: - source_labels: [__meta_dockerswarm_node_address, job]...
2
votes
0answers
26 views

Grafana K8s microservices availability dashboard based on actuator uptime

I need to create a dashboard that is using micrometer prometheus spring actuator endpoint to show all services for a given namespaces in green boxes or red boxes depending on status, the boxes should ...
0
votes
0answers
17 views

Docker Container CPU usage Monitoring

As per the documentation of docker. We can get CPU usage of docker container with docker stats command. The column CPU % will give the percentage of the host’s CPU the container is using. Let say I ...
0
votes
0answers
98 views

Spring boot prometheus - difference between system_cpu_usage and process_cpu_usage

Spring boot reports two statistics when prometheus in place. # HELP system_cpu_usage The "recent cpu usage" for the whole system # TYPE system_cpu_usage gauge system_cpu_usage 0....
1
vote
1answer
23 views

How to append namespace before metric name in prometheus?

I have static config in prometheus.yml - job_name: 'myjob' static_configs: - targets: ['monitoring.api.endpoint.net'] And I recieve various metrics for example cpu_usage, disk_write_ops, ...
0
votes
0answers
9 views

TokenAuth to monitor (or scrape) data from an API (Carbonite Backup)

I am currently exploring the best way to visualizing data from Carbonite Server Backup. They offer an simple Api that returns Json data. I want to visualize this data in Grafana. However the Api uses ...
0
votes
0answers
16 views

Get key value from a go template in Prometheus/Alertmanager/Jiralert attempt to map severity to Jira task

I am using Prometheus Alertmanager integration with Jiralert go tool to create tasks out of alerts in Jira. Everything works as expected aside from the fact that I don't know how to inherit severity ...
0
votes
0answers
12 views

How to edit Prometheus configuration of “Prometheus & Grafana” application deployed from the GKE Marketplace?

I am not sure if there is documentation about it or should it be self-explanatory but having deployed Prometheus and Grafana via the Google Kubernetes Engine marketplace in Google Cloud Platform, I am ...
1
vote
1answer
22 views

Is there a way to make Prometheus to scrape only the metrics which are present in Grafana?

As a customer of Grafna Cloud, I need to minimize the number of DataSeries that are being sent (remote_write) to Grafana Cloud, to avoid being overcharged for metrics that I don't really use. This can ...
3
votes
1answer
91 views

Prometheus Alertamanger - how to silence all alerts for a given period during a maintenance?

A work scenario that I can't cover currently is that I want to set a maintenance mode, meaning all alerts received from Prometheus to be ignored, I want to be able to set it through the UI for a given ...
0
votes
0answers
89 views

Confluent Kafka JMX curl on localhost - only default jmx response with no metrics from kafka

Running confluent helm chart 0.5.0, JMX is enabled in chart, when I login into broker and I curl localhost or 127.0.0.1 on port 5556 I get only some default endpoints and no metrics from kafka, why? ...
3
votes
1answer
26 views

Initial Configuration: Grafana and Prometheus inside docker and then AWS

I'm a bit of a newbie with this, but I haven't been able to find an explanation. Here's my issue. I'm trying to get Prometheus and Grafana configured on an AWS Fargate instance. I'm testing inside a ...
-1
votes
1answer
43 views

K8s to Prometheus apiserver_client_certificate_expiration_seconds_count metric, why does it count more seconds than 86400 for a single day?

In the 2 pictures below I am comparing the metric from K8s apiserver_client_certificate_expiration_seconds_count{job="apiserver"} for my 3 master nodes on K8s 1.14.3. A single day should have 86400 ...
1
vote
1answer
36 views

Reuse Alertmanger Slack Integration into Jiralert template

What kind of changes need to be made to a slack template in order to reuse into Jiralert one? Can I utilize the whole body as it is, can somebody post an example of a working resuse, even for the ...
0
votes
1answer
47 views

Disable creation of labels in Jira?

Following the base instructions I can't create new Jira item through API request due to failing because of trying to create labels. Even if I am using add_group_labels: false it tries to create label ...
5
votes
2answers
359 views

Back-filling prometheus (and related system) metrics?

Projects like Thanos, which are based on prometheus storage formats, seem to say that it is impossible to back-fill data in a prometheus based system. Can anyone explain why and if there are any ...
2
votes
0answers
76 views

Thanos or Cortex - What handles very large scale (say, hundreds of millions of time series) better?

We're looking at a new metrics solution and are attempting to build it in house. So, my question is: What can scale more effectively / with less pain; Thanos or Cortex? I understand the general ...
1
vote
0answers
23 views

Add a custom label to a metric when querying from prometheus/prometheus-operator

I am trying to make a view in Grafana that would return the result of node_memory_MemFree_bytes{job="node-exporter"}} < 1000000000 by nodes from Prometheus datasource (from Prometheus-operator ...
1
vote
1answer
28 views

Pometheus-operator - modify Grafana mixin dashboard

Whenever I try to edit the provisioned dashboards I get an error stating that the provisioned dashboard can't be edited. I saw that there is an option to allow changes from the UI, but my questions is ...
0
votes
0answers
57 views

In K8s - Prometheus, which endpoint to scrape from to get container_cpu_cfs_throttled_seconds_total metric

Scenario I have an Amazon K8s cluster (EKS) and am starting out with Prometheus & K8s monitoring. It seems to me there are so many endpoints to scrape from. Here are what I have applied: node-...
1
vote
1answer
278 views

Prometheus - join series based on the key part of a label

In 1.16 k8s deprecated the label kubernetes.io/role: worker I had previously been using as a filter to target graphs to specific node role using this in my grafana variables declaration label_values(...
0
votes
1answer
264 views

Prometheus-operator - detection of expiring certs failing

I had a situation yesterday where my K8S master api cert has expired and I had Prometheus-operator active in the past weeks, but neither the 24 hours, neither the 7 days alert has triggered. I went to ...
4
votes
2answers
102 views

What is the Prometheus and Grafana ideal setup?

I am just wondering. If I have many environments monitored via Prometheus, what will be the best configuration? The security and efficiency are already important for on-premise installations but ...
1
vote
1answer
242 views

What is Unit testing in the context of YML configuration Alerting rules?

I read a very short blog posting about Unit testing alerts with Prometheus Can you explain me why I need unit testing for *.yml? What is the benefit of unit testing for configuration files? I not ...
0
votes
0answers
60 views

How to use the consul_io.py dynamic inventory script for grouping and for proper labelling in prometheus?

I've configured service-discovery in Consul for on-premiss servers in the company I work for and I'm using consul_io.py as a dynamic inventory script for Ansible. When I run the script, I get the ...
4
votes
1answer
125 views

Using Prometheus to monitor Spring Boot Applications in Kubernetes Cluster

I have spring boot powered microservices deployed in my local kubernetes cluster. The microservices are using micrometer and prometheus registry but due to our company policy the actuator is available ...
2
votes
0answers
139 views

Grafana sometimes can't resolve prometheus hostname [closed]

Scenario I deploy grafana and prometheus onto EKS cluster (AWS K8s service). If I use prometheus service's fqdn (prometheus-server.monitoring.svc.cluster.local) as the data source, grafana sometimes ...
0
votes
1answer
182 views

Replace/Consume Istio's Prometheus/Grafana by existing Prometheus-operator

I have multiple K8S clusters that have Prometheus-operator Helm chart installed in monitoring namespace. Each operator consists of Prometheus/Alertmanager/Grafana. Aside from it I have Istio Helm ...
1
vote
2answers
38 views

Gather resource usage by process in a kubernetes cluster

I am searching for a tool similar to Prometheus + Grafana that gather and record resource usage especially memory usage by process-ID or process-name. We have two components that are running ...
4
votes
1answer
3k views

How to calculate disk space required by Prometheus v2.2?

We are trying to calculate the storage requirements but is unable to find the values needed to do the calculation for our version of Prometheus (v2.2). Prometheus(v2.2) storage documentation gives ...
1
vote
1answer
109 views

Expose metrics for Kotlin using JMX_Exporter to prometheus

I am trying to use JMX_Exporter for my kotlin code to expose metrics to prometheus in order to show it in Grafana. I have gone through many articles and tried to understand how could it be done. I ...
5
votes
1answer
1k views

How to calculate burn rate for SLOs?

I've read the Google SRE book a few times but I need some clarifications on exactly how to set up the burn rate and understanding how long it'll take to trigger an alert. Most of my questions are ...
1
vote
2answers
82 views

What is the advantage of using Prometheus over ingesting metrics using logs?

I want to setup monitoring on our environment. I am wondering on what are the advantages of using something like Prometheus for making metrics vs. just using something like the ELK stack for creating ...
7
votes
1answer
3k views

Why is Prometheus not a good choice for data with high cardinality?

I have a background in relational databases and new to Prometheus. I wonder why Prometheus is not a good choice for high cardinality data? Why do I need to use low cardinality data? It's exact ...
4
votes
1answer
5k views

Prometheus alert CPUThrottlingHigh raised but monitoring does not show it

I have installed Prometheus to monitor my installation and it is frequently raising alerts about CPU throttling. The Prometheus alert rules to identify this alert is : alert: CPUThrottlingHigh expr: ...
2
votes
1answer
32 views

Which libraries generate observability metrics for applications in the cluster to prometheus?

I recently read "Cloud Native DevOps with Kubernetes" by O'reily and it mentions the RED and USE metrics which are a good starting point for gaining better observability into the system, however I'm ...
2
votes
1answer
498 views

Keeping graphs of terminated Kubernetes pods in Prometheus/Grafana

I love using Prometheus/Grafana to monitor and tune resource usage on my Kubernetes clusters. When pods are terminated, however, the graphs are disposed of it seems. I am looking for a solution to ...
4
votes
1answer
221 views

Does Prometheus expose the Horizontal Pod Autoscaler's “Current CPU Utilization” as shown in the Kubernetes dashboard?

In the Kubernetes dashboard, I can see for a HPA the following information: Min Replicas: 3 Max Replicas: 11 Target CPU Utilization: 80% Status Current Replicas: 3 Desired Replicas: 3 Current CPU ...
1
vote
0answers
55 views

How do I set up a Prometheus rule to routinely scrape an HTTP endpoint? [closed]

I'm getting started with Prometheus (Prometheus Operator), and I'm looking at adding it to an existing cluster, where each container has a /healthcheck endpoint, that is going to either return true, ...
1
vote
2answers
245 views

Jenkins plugin recomendation

My company wants to visualize the % of failed/successful deployments of jenkins jobs. And the way jenkins is arranged complicates the task, each team or project is a folder name and inside the ...
4
votes
1answer
2k views

How fast does Prometheus data grow?

This may sound like a vague question so I will provide some context below. The basic question is : What parameters describe the growth in size of a Prometheus database over time? To tie the question ...