Questions tagged [prometheus]

For questions about Prometheus, an open-source systems monitoring and alerting toolkit.

Filter by
Sorted by
Tagged with
0 votes
1 answer
33 views

Prometheus/BlackBox Exporter probe_success 0 but service still running

I'm currently utilizing Prometheus with Blackbox Exporter for monitoring our services. However, I've encountered an issue where one of our external services is showing probe_success as 0, despite ...
AwsmOfficial's user avatar
0 votes
0 answers
13 views

Setup TLS Certs for Prometheus adapter in Local k3d

I am trying to setup HorizontalPodAutoscaler in K8s, for which I am setting up a custom metric to be used by HPA. I am using k3d locally to test the autoscaling feature. For this custom metric , I ...
devcodes's user avatar
  • 111
1 vote
1 answer
41 views

Add prometheus config for all jobs

In my prometheus.yml I have several jobs defined like: - job_name: alpha scheme: 'https' metrics_path: '/metrics' static_configs: - targets: - one.example - two....
Vietna's user avatar
  • 25
0 votes
0 answers
13 views

Write_Prometheus plugin with basic auth

0 I am testing the below Write_Prometheus plugin to send the cassandra metrics to prometheus server which is external machine but i see that all metrics are sending without any basic auth happening. ...
rohit singhal's user avatar
0 votes
0 answers
47 views

Why Kube state metrics only shows metrics related to the namespace where it is running?

I have AWS EKS cluster with kube-state-metrics installed in a namespace called "monitoring". This installation is using service monitor and other components (see yaml files below). In this ...
Thiago Scodeler's user avatar
0 votes
1 answer
35 views

How to scrape only some targets of EC2 based on tag

I have a requirement, where I have enabled NGINX metrics and I am trying to plot metrics with Prometheus. I am trying to plot metrics for NGINX EC2 server which has a tag named test=nginx or by ...
Siva Malla's user avatar
0 votes
0 answers
9 views

Can I use statsD to scrape Kubernetes metric endpoint or from Kubernetes metric exporters?

Can I use statsD to scrape Kubernetes metric endpoint or from Kubernetes metric exporters(for kubernetes infrastructure metrics like nodes, pods, control plane)? I see people are using Prometheus more....
bunny's user avatar
  • 101
0 votes
0 answers
23 views

Is there a way to send alert emails if Kubernetes detects an authentication attempt, even if I don't have access to the master node?

I have a Kubernetes cluster in OVH, and I need to send alerts if the API detects an authentication attempt. I know I could use a script to monitor Kubernetes audit log files and send emails if there ...
M.JOUAHER's user avatar
1 vote
1 answer
24 views

get node_exporter metrics for a single system user

I have installed the node_exporter within the userspace of a linux system user. Everything works and I get the metrics in my grafana. But what I get is apperently the metrics for the whole host system ...
Vietna's user avatar
  • 25
3 votes
1 answer
21 views

Prometheus and Azure WebApps: How to monitor multiple instances with prometheus

I have an application that has two instances running and each request runs on a different machine. I am monitoring an event with a counter in my application and each instance runs it's own counter ...
Menelaos Vergis's user avatar
1 vote
1 answer
329 views

Prometheus scrape from service vs pods

Let's say I have multiple pods (replicas) of application under a service (with targetPort pointing to port where application keeps its metrics for Prometheus) named "app-monitor-service". ...
user avatar
0 votes
1 answer
57 views

Aws EKS Prometheus helm failed pod CrashLoopBackOff with log "trigger reload: reload request failed: Post \"http://127.0.0.1:9090/-/reload\"

I followed the steps here https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-onboard-ingest-metrics-new-Prometheus.html In a nutshell, I installed the Prometheus community helm chart on my ...
Harry's user avatar
  • 101
0 votes
1 answer
105 views

Metrics Server and Prometheus have overwriting resources

I am trying to install Metrics Server, but it needs to create a resource that already exists from an installation of Prometheus. They both want to make v1beta1.metrics.k8s.io. I went to see if the ...
Vaccano's user avatar
  • 206
0 votes
0 answers
8 views

How do I visualize multiple static Prometheus TSDB files from a single Grafana instance?

I have a requirement in my project where I receive multiple static TSDB files which are essentially Prometheus backups and then visualize these on Grafana using dashboards. I am looking to understand ...
PunitK's user avatar
  • 1
0 votes
0 answers
23 views

How to configure a prometheus installation of openshift to include the labels of node role in the exporter?

We want to alert based on cpu requests only on worker nodes namespace_cpu:kube_pod_container_resource_requests:sum{kubernetes_node_role=~"worker"} gives no datapoints and namespace_cpu:...
Adrián Blázquez's user avatar
0 votes
0 answers
181 views

Unexpected response status when fetching server time error in thanos query dashboard

I have installed the "monitoring" app (via rancher UI) which is provided by kubernetes. On top of it, I have deployed thanos-sidecar as 4th container in the prometheus pod running in cattle-...
anonymous user's user avatar
1 vote
0 answers
79 views

Can Loki accept/combine a log stream (unique set of labels) from multiple Promtail instances running simultaneously? (applies to Prometheus too)

I am using ec2_sd_configs in Loki (and Prometheus) to discover my EC2 instances, most of which are autoscaled, and therefore come and go. They are identical load balanced API instances, so I don't ...
Life5ign's user avatar
  • 130
3 votes
0 answers
70 views

Measuring image pull latency in k8s cluster

I maintain a Kubernetes clusters that pulls a number of images (mostly from Docker Hub, but also some from GCR and GHCR). I am interested in building a dashboard to measure the amount of time it takes ...
Derek Brown's user avatar
1 vote
0 answers
28 views

Best way to rollout open telemetry helm operators and resources in a multi-cluster K8s Setup

I am currently tasked with building an observability pipeline based on open telemetry. We are looking at shipping all 3 key signals (metrics, traces, and logs) to the Grafana cloud. We also have a ...
sage's user avatar
  • 199
0 votes
2 answers
313 views

How to defragment etcd database in Kubernetes cluster

I am trying to resolve a recurring Prometheus alert I'm getting that states that my Kubernetes cluster etcd database is fragmented and should be defragmented. I have never used the etcdctl command ...
Joseph Gagnon's user avatar
0 votes
1 answer
130 views

List all active promtail targets

I have spend a significant time reading the manual, but I still cannot figure out how to actually get a list of the targets that promtail is currently tailing. The metrics endpoint only exposes ...
MrMeeSeeks's user avatar
0 votes
0 answers
168 views

Prometheus HTTP API returns no metrics data for valid target

I have a target in Prometheus that's functional and is polling successfully; Grafana is able to see the data from Prometheus and render appropriate charts/dashboards. However, when I poll the HTTP API ...
Numpty's user avatar
  • 101
0 votes
1 answer
575 views

Add more static instance labels in prometheus config

I'm adding serves to my prometheus config like this: - job_name: cluster1 static_configs: - targets: - server1.example.com - server2.example.com - server3.example....
Vietna's user avatar
  • 25
0 votes
1 answer
32 views

Enterprise Ready Prometheus

I am new to Prometheus and have been reading about its popularity as a tool for monitoring applications, containers and K8s objects. However, most of the blogs that I have come across have at best ...
Gaurav Parashar's user avatar
0 votes
0 answers
86 views

How to calculate Jenkins Job increase in build time through Prometheus query

I extract statistics from Jenkins with the Prometheus Metrics plugin. I have created a query in PromQL to check if a Jenkins job builds time has increased by 50% from the average successful build time:...
Thanos Mourtzoukos's user avatar
1 vote
1 answer
317 views

Prometheus scrape config for kubelet node role not auth'ing

I have a kubernetes (not minikube) cluster (1.25) installed on a dedicated machine (1-node cluster). I installed Prometheus. I am using a ServiceAccount, Role, ClusterRoleBinding like this: apiVersion:...
codedread's user avatar
  • 111
1 vote
1 answer
193 views

How to add an aditional ServiceMonitor for prometheus-operator?

I installed prometheus-operator using kube-prometheus-stack and so far everything has worked fine, but I need to add one more service to monitoring. This is a Windows Server that is not running on the ...
timeuz's user avatar
  • 11
2 votes
0 answers
37 views

Why is an 'average' as a metric favored (over 'latency') in this case?

In a talk on RED method, the presenter defends his use of 'average' by the following comparison with latencies. The linked-to part of the talk written down (non-verbatim and with my formatting): ...
VanillaDonuts's user avatar
1 vote
0 answers
169 views

Can't access default k3s metrics server from Prometheus-Adapter

I have installed Prometheus-adapter along with the default metrics-server that comes with k3s securely on port 4443. Unfortunately, I get no resources when I query custom.metrics.k8s.io $ kubectl get ...
realsarm's user avatar
  • 111
3 votes
0 answers
158 views

How to collect statistics from CI over time?

CI environments like GitLab (self-hostet and Cloud), GitHub, CodeCov, Codacy, ... collect statistics over time, so a developer or team lead can see the evolution of the project over time: number of ...
Paebbels's user avatar
  • 131
0 votes
1 answer
505 views

Kubernetes Job Metrics in Prometheus

I want to create a Kubernetes Job object like the example: https://kubernetes.io/docs/concepts/workloads/controllers/job/ Now imagine I want to have the result (in this case the value of Pi: 3.14159......
Ezariah A's user avatar
0 votes
0 answers
927 views

basic authentication prometheus pod

I'm trying to add basic authentication for prometheus pod but when I deploy the pod with the below config I get the error: "Readiness probe failed: HTTP probe failed with statuscode: 401", I ...
Petrisor Petrisor's user avatar
0 votes
1 answer
570 views

How to purge / prevent using the prometheus local storage when using remote_write

I configured Prometheus to push metrics using remote_write but the local storage is still in use and keeps growing How could Prometheus be configured to stop storing locally and only push metrics or ...
nbari's user avatar
  • 101
0 votes
1 answer
224 views

Snapshot or Saving Prometheus Data for Performance testing and comparison

I have an application and system that exposes its resource metrics to Prometheus and Grafana that I want to do performance testing on. I want to be able to conduct a test, capture the metrics on CPU, ...
DogEatDog's user avatar
  • 191
0 votes
0 answers
2k views

How do I delete ALL time series in prometheus without restarting?

How do I delete all data? I tried: Using the HTTP API. Deleting a series with a match operator such as curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={job="BR&...
Nearoo's user avatar
  • 101
3 votes
1 answer
3k views

What is the difference between Fluent Bit and Prometheus?

From what I've seen, Prometheus scrapes data and metrics and displays them and Fluent Bit collects logs and metrics and can output them. So, are they doing the same thing, i.e. should I choose one ...
Grafana's user avatar
  • 33
0 votes
1 answer
90 views

Apache Camel-k metrics in rancher embedded prometheus

I have a Kubernetes running Rancher 2.4.3. I have the cluster monitoring enabled in rancher, so that exists a Prometheus instance running, so as a Prometheus Operator. Recently I deployed a Apache ...
Beto Neto's user avatar
  • 101
2 votes
1 answer
2k views

Does Prometheus log scrape-duration?

I'd like to graph scrape duration for a particular target. The Prometheus web interface has a Status of Targets page, which tabulates the (time elapsed since) "last scrape" and corresponding ...
benjimin's user avatar
  • 269
0 votes
1 answer
439 views

Scraping different Prometheus metrics at different rates

I am running a Tendermint network. I am using Prometheus to track various metrics, and to track node health. For performance metrics such as tendermint_consensus_total_txs, I want updates every second....
Ethan Reesor's user avatar
3 votes
0 answers
807 views

Create a Kubernetes Service that loads an external host url AND port

I have a url like this: http://myserver.mydomain.com:98765 I need to have this url be a Kubernetes service so I can address it in Kubernetes like this: my-service.mynamespace.svc.cluster.local. (So, ...
Vaccano's user avatar
  • 206
2 votes
2 answers
3k views

Monitoring Kubernetes CronJob & Job in Grafana for today day only

I'm trying to create a Dashboard in Grafana that will show the status of Cronjob & Job success & fail. Currently i'm using kube_job_status_succeeded to get a status job, but the issue here ...
DagaReiN's user avatar
2 votes
1 answer
146 views

How prometheus scrapes individual pods actuator endpoints when they are exposed via a single service in kubernetes?

I have created a deployment with 2 replica sets. But the Service exposes the 8080 and the management endpoint using NodePort. How should I set up the pods deployment if Prometheus deployment should ...
GauravEdekar's user avatar
0 votes
1 answer
517 views

How to setup basic auth for Prometheus deployed on K8s cluster using yamls?

How to setup basic auth for Prometheus deployed on K8s cluster using yamls?? I was able to achieve this easily when Prometheus was deployed on a host locally using tar file. But when it is deployed as ...
Saurabh's user avatar
1 vote
2 answers
4k views

Enable ingress-nginx Prometheus monitoring without Helm

A few weeks ago, I installed ingress-nginx in my Kubernetes cluster. Now, I realize that was a mistake because that did not come with Prometheus monitoring enabled automatically, and even worse, it ...
chrispytoes's user avatar
1 vote
0 answers
990 views

Unable to fetch metrics from custom metrics API: Internal error occurred

I am running a local Kubernetes cluster with Minikube on a Windows machine using --driver=hyperv. I am trying to set up an HPA (HorizontalPodAutoscaler) based on Pods custom metrics. Using Helm, I ...
soders's user avatar
  • 11
0 votes
2 answers
762 views

Getting error while trying to communicate with apiserver in kube-state-metrics pod

I am getting error: # kubectl logs kube-state-metrics-fdc49658-g98sl -n monitoring I0523 11:30:25.206556 1 main.go:86] Using default collectors I0523 11:30:25.206766 1 main.go:98] Using ...
Chang Zhao's user avatar
1 vote
1 answer
348 views

Where to install Prometheus for Kubernetes monitoring

I'm considering using Prometheus to monitor and analyze different metrics from my AWS EKS Kubernetes cluster running in Fargate, as it seems to be the standard tool during these times. However, ...
Nicolás García's user avatar
2 votes
1 answer
3k views

How to filter label_values in Loki?

I want to create a query variable in grafana using Loki that returns all the possible label values of the label baz such that they only include label values that exist where the label foo equals "...
user2324712's user avatar
0 votes
1 answer
428 views

Node-centric network metrics are very different from the sum of metrics of the Pods running on that node

Context I am trying to chart the network bandwidth usage of a node in 2 different manners: By looking at global metrics for that node By summing up the corresponding metric for each Pod To achieve ...
Antoine Cotten's user avatar
0 votes
1 answer
167 views

micrometer exposing actuator metrics vs kube-state-metrics vs metrics-server to set pod request/limits

micrometer exposing actuator metrics to set request/limit to pods in K8svs metrics-server vs kube-state-metrics -> K8s Mixin from kube-promethteus-stack Grafana dashboad It's really blurry and ...
anVzdGFub3RoZXJodW1hbg's user avatar