Questions tagged [prometheus]

For questions about Prometheus, an open-source systems monitoring and alerting toolkit.

Filter by
Sorted by
Tagged with
0 votes
0 answers
18 views

Loki-stack helm chart is not allowing to edit grafana.ini

I have deployed Loki-stack using helm charts. For configuring email alerts, I need to modify grafana.ini. I am not able to edit grafana.ini as it has read-only permissions. Even I could find a way to ...
user avatar
0 votes
0 answers
55 views

basic authentication prometheus pod

I'm trying to add basic authentication for prometheus pod but when I deploy the pod with the below config I get the error: "Readiness probe failed: HTTP probe failed with statuscode: 401", I ...
user avatar
0 votes
0 answers
19 views

Prometheus not discovering Kubernetes service based on type and port

I have a Kubernetes cluster that e.g. hosts service/my-service with the following key characteristics: apiVersion: v1 kind: Service metadata: name: my-service namespace: my-namespace spec: ports:...
user avatar
  • 141
0 votes
1 answer
21 views

How to purge / prevent using the prometheus local storage when using remote_write

I configured Prometheus to push metrics using remote_write but the local storage is still in use and keeps growing How could Prometheus be configured to stop storing locally and only push metrics or ...
user avatar
  • 101
0 votes
1 answer
42 views

Snapshot or Saving Prometheus Data for Performance testing and comparison

I have an application and system that exposes its resource metrics to Prometheus and Grafana that I want to do performance testing on. I want to be able to conduct a test, capture the metrics on CPU, ...
user avatar
  • 161
0 votes
0 answers
241 views

How do I delete ALL time series in prometheus without restarting?

How do I delete all data? I tried: Using the HTTP API. Deleting a series with a match operator such as curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={job="BR&...
user avatar
  • 101
1 vote
1 answer
359 views

What is the difference between Fluent Bit and Prometheus?

From what I've seen, Prometheus scrapes data and metrics and displays them and Fluent Bit collects logs and metrics and can output them. So, are they doing the same thing, i.e. should I choose one ...
user avatar
  • 13
0 votes
1 answer
34 views

Apache Camel-k metrics in rancher embedded prometheus

I have a Kubernetes running Rancher 2.4.3. I have the cluster monitoring enabled in rancher, so that exists a Prometheus instance running, so as a Prometheus Operator. Recently I deployed a Apache ...
user avatar
  • 101
0 votes
1 answer
137 views

Does Prometheus log scrape-duration?

I'd like to graph scrape duration for a particular target. The Prometheus web interface has a Status of Targets page, which tabulates the (time elapsed since) "last scrape" and corresponding ...
user avatar
  • 113
0 votes
1 answer
61 views

Scraping different Prometheus metrics at different rates

I am running a Tendermint network. I am using Prometheus to track various metrics, and to track node health. For performance metrics such as tendermint_consensus_total_txs, I want updates every second....
user avatar
2 votes
0 answers
129 views

Create a Kubernetes Service that loads an external host url AND port

I have a url like this: http://myserver.mydomain.com:98765 I need to have this url be a Kubernetes service so I can address it in Kubernetes like this: my-service.mynamespace.svc.cluster.local. (So, ...
user avatar
  • 172
1 vote
2 answers
706 views

Monitoring Kubernetes CronJob & Job in Grafana for today day only

I'm trying to create a Dashboard in Grafana that will show the status of Cronjob & Job success & fail. Currently i'm using kube_job_status_succeeded to get a status job, but the issue here ...
user avatar
0 votes
0 answers
21 views

Does Thanos' object storage requires internet connection to work?

I'm considering to use Thanos as a long-term storage for Prometheus. For that Thanos requires to use an object storage such as Amazon S3/AWS, Google Cloud etc. I have no experience with cloud services,...
user avatar
  • 1
1 vote
1 answer
56 views

How prometheus scrapes individual pods actuator endpoints when they are exposed via a single service in kubernetes?

I have created a deployment with 2 replica sets. But the Service exposes the 8080 and the management endpoint using NodePort. How should I set up the pods deployment if Prometheus deployment should ...
user avatar
0 votes
1 answer
80 views

How to setup basic auth for Prometheus deployed on K8s cluster using yamls?

How to setup basic auth for Prometheus deployed on K8s cluster using yamls?? I was able to achieve this easily when Prometheus was deployed on a host locally using tar file. But when it is deployed as ...
user avatar
1 vote
2 answers
1k views

Enable ingress-nginx Prometheus monitoring without Helm

A few weeks ago, I installed ingress-nginx in my Kubernetes cluster. Now, I realize that was a mistake because that did not come with Prometheus monitoring enabled automatically, and even worse, it ...
user avatar
1 vote
0 answers
399 views

Unable to fetch metrics from custom metrics API: Internal error occurred

I am running a local Kubernetes cluster with Minikube on a Windows machine using --driver=hyperv. I am trying to set up an HPA (HorizontalPodAutoscaler) based on Pods custom metrics. Using Helm, I ...
user avatar
  • 11
0 votes
1 answer
321 views

Getting error while trying to communicate with apiserver in kube-state-metrics pod

I am getting error: # kubectl logs kube-state-metrics-fdc49658-g98sl -n monitoring I0523 11:30:25.206556 1 main.go:86] Using default collectors I0523 11:30:25.206766 1 main.go:98] Using ...
user avatar
1 vote
1 answer
160 views

Where to install Prometheus for Kubernetes monitoring

I'm considering using Prometheus to monitor and analyze different metrics from my AWS EKS Kubernetes cluster running in Fargate, as it seems to be the standard tool during these times. However, ...
user avatar
2 votes
1 answer
2k views

How to filter label_values in Loki?

I want to create a query variable in grafana using Loki that returns all the possible label values of the label baz such that they only include label values that exist where the label foo equals "...
user avatar
0 votes
1 answer
114 views

Node-centric network metrics are very different from the sum of metrics of the Pods running on that node

Context I am trying to chart the network bandwidth usage of a node in 2 different manners: By looking at global metrics for that node By summing up the corresponding metric for each Pod To achieve ...
user avatar
0 votes
1 answer
64 views

micrometer exposing actuator metrics vs kube-state-metrics vs metrics-server to set pod request/limits

micrometer exposing actuator metrics to set request/limit to pods in K8svs metrics-server vs kube-state-metrics -> K8s Mixin from kube-promethteus-stack Grafana dashboad It's really blurry and ...
user avatar
1 vote
2 answers
175 views

Is it possible/fine to run Prometheus, Loki, Grafana outside of Kubernetes?

In some project there are scaling and orchestration implemented using technologies of a local cloud provider, with no Docker & Kubernetes. But the project has poor logging and monitoring, I'd like ...
user avatar
  • 181
0 votes
1 answer
57 views

Grafana Status Panel Plugin - genereate status box for each entry from a Prometheus querry

I am trying to create a dashboad out of process_uptime_seconds{namespace="$namespace"} querry from Spring Actuator. and Prometheus as datasource I am trying to build a single stat box menus ...
user avatar
3 votes
0 answers
90 views

Grafana K8s microservices availability dashboard based on actuator uptime

I need to create a dashboard that is using micrometer prometheus spring actuator endpoint to show all services for a given namespaces in green boxes or red boxes depending on status, the boxes should ...
user avatar
2 votes
1 answer
2k views

Spring boot prometheus - difference between system_cpu_usage and process_cpu_usage

Spring boot reports two statistics when Prometheus in place. # HELP system_cpu_usage The "recent cpu usage" for the whole system # TYPE system_cpu_usage gauge system_cpu_usage 0....
user avatar
  • 243
1 vote
1 answer
68 views

How to append namespace before metric name in prometheus?

I have static config in prometheus.yml - job_name: 'myjob' static_configs: - targets: ['monitoring.api.endpoint.net'] And I recieve various metrics for example cpu_usage, disk_write_ops, ...
user avatar
1 vote
1 answer
69 views

Is there a way to make Prometheus to scrape only the metrics which are present in Grafana?

As a customer of Grafna Cloud, I need to minimize the number of DataSeries that are being sent (remote_write) to Grafana Cloud, to avoid being overcharged for metrics that I don't really use. This can ...
user avatar
4 votes
2 answers
5k views

Prometheus Alertamanger - how to silence all alerts for a given period during a maintenance?

A work scenario that I can't cover currently is that I want to set a maintenance mode, meaning all alerts received from Prometheus to be ignored, I want to be able to set it through the UI for a given ...
user avatar
3 votes
1 answer
54 views

Initial Configuration: Grafana and Prometheus inside docker and then AWS

I'm a bit of a newbie with this, but I haven't been able to find an explanation. Here's my issue. I'm trying to get Prometheus and Grafana configured on an AWS Fargate instance. I'm testing inside a ...
user avatar
-1 votes
1 answer
194 views

K8s to Prometheus apiserver_client_certificate_expiration_seconds_count metric, why does it count more seconds than 86400 for a single day?

In the 2 pictures below I am comparing the metric from K8s apiserver_client_certificate_expiration_seconds_count{job="apiserver"} for my 3 master nodes on K8s 1.14.3. A single day should have 86400 ...
user avatar
1 vote
1 answer
111 views

Reuse Alertmanger Slack Integration into Jiralert template

What kind of changes need to be made to a slack template in order to reuse into Jiralert one? Can I utilize the whole body as it is, can somebody post an example of a working resuse, even for the ...
user avatar
0 votes
1 answer
161 views

Disable creation of labels in Jira?

Following the base instructions I can't create new Jira item through API request due to failing because of trying to create labels. Even if I am using add_group_labels: false it tries to create label ...
user avatar
5 votes
2 answers
1k views

Back-filling prometheus (and related system) metrics?

Projects like Thanos, which are based on prometheus storage formats, seem to say that it is impossible to back-fill data in a prometheus based system. Can anyone explain why and if there are any ...
user avatar
3 votes
0 answers
393 views

Thanos or Cortex - What handles very large scale (say, hundreds of millions of time series) better?

We're looking at a new metrics solution and are attempting to build it in house. So, my question is: What can scale more effectively / with less pain; Thanos or Cortex? I understand the general ...
user avatar
1 vote
1 answer
45 views

Pometheus-operator - modify Grafana mixin dashboard

Whenever I try to edit the provisioned dashboards I get an error stating that the provisioned dashboard can't be edited. I saw that there is an option to allow changes from the UI, but my questions is ...
user avatar
1 vote
1 answer
3k views

Prometheus - join series based on the key part of a label

In 1.16 k8s deprecated the label kubernetes.io/role: worker I had previously been using as a filter to target graphs to specific node role using this in my grafana variables declaration label_values(...
user avatar
0 votes
1 answer
1k views

Prometheus-operator - detection of expiring certs failing

I had a situation yesterday where my K8S master api cert has expired and I had Prometheus-operator active in the past weeks, but neither the 24 hours, neither the 7 days alert has triggered. I went to ...
user avatar
4 votes
3 answers
174 views

What is the Prometheus and Grafana ideal setup?

I am just wondering. If I have many environments monitored via Prometheus, what will be the best configuration? The security and efficiency are already important for on-premise installations but ...
user avatar
  • 155
1 vote
1 answer
786 views

What is Unit testing in the context of YML configuration Alerting rules?

I read a very short blog posting about Unit testing alerts with Prometheus Can you explain me why I need unit testing for *.yml? What is the benefit of unit testing for configuration files? I not ...
user avatar
  • 452
4 votes
1 answer
473 views

Using Prometheus to monitor Spring Boot Applications in Kubernetes Cluster

I have spring boot powered microservices deployed in my local kubernetes cluster. The microservices are using micrometer and prometheus registry but due to our company policy the actuator is available ...
user avatar
4 votes
0 answers
641 views

Grafana sometimes can't resolve prometheus hostname [closed]

Scenario I deploy grafana and prometheus onto EKS cluster (AWS K8s service). If I use prometheus service's fqdn (prometheus-server.monitoring.svc.cluster.local) as the data source, grafana sometimes ...
user avatar
0 votes
1 answer
265 views

Replace/Consume Istio's Prometheus/Grafana by existing Prometheus-operator

I have multiple K8S clusters that have Prometheus-operator Helm chart installed in monitoring namespace. Each operator consists of Prometheus/Alertmanager/Grafana. Aside from it I have Istio Helm ...
user avatar
1 vote
2 answers
63 views

Gather resource usage by process in a kubernetes cluster

I am searching for a tool similar to Prometheus + Grafana that gather and record resource usage especially memory usage by process-ID or process-name. We have two components that are running ...
user avatar
  • 205
9 votes
2 answers
9k views

How to calculate disk space required by Prometheus v2.2?

We are trying to calculate the storage requirements but is unable to find the values needed to do the calculation for our version of Prometheus (v2.2). Prometheus(v2.2) storage documentation gives ...
user avatar
1 vote
1 answer
180 views

Expose metrics for Kotlin using JMX_Exporter to prometheus

I am trying to use JMX_Exporter for my kotlin code to expose metrics to prometheus in order to show it in Grafana. I have gone through many articles and tried to understand how could it be done. I ...
user avatar
  • 121
7 votes
1 answer
3k views

How to calculate burn rate for SLOs?

I've read the Google SRE book a few times but I need some clarifications on exactly how to set up the burn rate and understanding how long it'll take to trigger an alert. Most of my questions are ...
user avatar
1 vote
2 answers
123 views

What is the advantage of using Prometheus over ingesting metrics using logs?

I want to setup monitoring on our environment. I am wondering on what are the advantages of using something like Prometheus for making metrics vs. just using something like the ELK stack for creating ...
user avatar
  • 13
8 votes
1 answer
6k views

Why is Prometheus not a good choice for data with high cardinality?

I have a background in relational databases and new to Prometheus. I wonder why Prometheus is not a good choice for high cardinality data? Why do I need to use low cardinality data? It's exact ...
user avatar
  • 452
7 votes
2 answers
13k views

Prometheus alert CPUThrottlingHigh raised but monitoring does not show it

I have installed Prometheus to monitor my installation and it is frequently raising alerts about CPU throttling. The Prometheus alert rules to identify this alert is : alert: CPUThrottlingHigh expr: ...
user avatar
  • 173