Questions tagged [prometheus]
For questions about Prometheus, an open-source systems monitoring and alerting toolkit.
90
questions
0
votes
0
answers
16
views
Difference between series cardinality on influxdb and prometheus
We are currently using influxdb 1.8 as our metrics store in our monitoring stack and are looking to see if the current stack can be moved to Prometheus.
One of the aspects that we are looking at is ...
- 473
0
votes
0
answers
64
views
How to install prometheus elasticsearch exporter by helm?
I installed Prometheus by
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install kube-prometheus-stack prometheus-community/kube-prometheus-...
- 101
0
votes
0
answers
42
views
How to setup EKS ADOT for sending OpenTelemetry metrics to self managed prometheus?
I have tried to configure ADOT collector using below yaml file
apiVersion: apps/v1
kind: Deployment
metadata:
name: adot-collector
namespace: monitoring
spec:
replicas: 1
selector:
...
0
votes
0
answers
87
views
How to get a label value from another metric and add it on prometheus
Suppose a metric is like the following,
weblogic_id(domain="name", instance="instance"}
Is there any way I can get the domain="name" from that metric and add it to all ...
- 1
0
votes
0
answers
22
views
Troubleshooting the 'View in Alertmanager' Button in MultiClusterObservability Email Notifications
This question concerns Openshift, but perhaps someone who simply deals with Kubernetes might be able to help. I have 2 clusters and on one of them I deployed Prometheus Alertmanager on Redhat ACM ...
- 1
1
vote
2
answers
727
views
Prometheus not Starting (Exec Format Error)
I am trying to run Prometheus on my Ubuntu Linux VM on a Mac M1 (if those details are necessary).
OS: Ubuntu 23.10
Prometheus: 2.51.2
I make the config files, and as I run the following lines,
sudo ...
- 11
0
votes
2
answers
553
views
How collect these metrics in Prometheus for an AWS ECS container?
We have a container running in ECS, and we want to collect the following metrics (truncated list) in Prometheus and eventually view them in Grafana. How do I setup the Prometheus agent on the ...
- 463
1
vote
1
answer
121
views
Dynamic service discovery for Docker containers in Prometheus configuration
I have an application running in a Docker stack, and it dynamically scales its instances based on demand. This application exports its own metrics, and I want Prometheus to scrape these metrics in ...
- 11
0
votes
0
answers
76
views
AWS EKS Fargate pod scheduling issue with Prometheus deployment
I'm encountering a problem with scheduling Prometheus pods on AWS EKS Fargate within my Kubernetes cluster. Here are the details of my setup:
I have configured an AWS EKS Fargate profile named ...
0
votes
1
answer
65
views
Concat multiple meta labels on prometheus for relabling
How can I change the value of the label __param_target to resemble how I would interact with a Kubernetes service in-cluster, following the format: http://serviceName.namespace.svc:servicePort/path ?
...
- 46
0
votes
0
answers
7
views
Remove trailing space in Mac OS hostname?
I'm trying to run Prometheus on my Mac, but I get the following
MyHostName $ prometheus --config.file=prometheus.yaml
parse external URL "": parse "http://MyHostName :9090/": ...
- 463
0
votes
0
answers
13
views
Question about AWS Prometheus tutorial
I'm following this cool example on how to run a container and monitor it CloudWatch with Prometheus - Monitor and scale your Amazon ECS on AWS Fargate application using Prometheus metrics. All my ...
- 463
0
votes
1
answer
558
views
How to configure MySQL exporter Helm chart for multiple targets (remote & local)?
I have been trying for a few days to set up a multi-target MySQL exporter (MSE) deployed with Helm inside a k8s cluster. The chart that I am using is the https://artifacthub.io/packages/helm/...
- 199
0
votes
1
answer
68
views
No targets in Prometheus at Kubernetes
Why are there no targets at my Prometheus? I did everything like in the Prometheus documentation. Also, I have pods with the label: "name=cadvisor".
I have the following configuration file:
#...
0
votes
1
answer
467
views
Prometheus/BlackBox Exporter probe_success 0 but service still running
I'm currently utilizing Prometheus with Blackbox Exporter for monitoring our services. However, I've encountered an issue where one of our external services is showing probe_success as 0, despite ...
1
vote
2
answers
746
views
Add prometheus config for all jobs
In my prometheus.yml I have several jobs defined like:
- job_name: alpha
scheme: 'https'
metrics_path: '/metrics'
static_configs:
- targets:
- one.example
- two....
- 25
0
votes
0
answers
19
views
Write_Prometheus plugin with basic auth
0
I am testing the below Write_Prometheus plugin to send the cassandra metrics to prometheus server which is external machine but i see that all metrics are sending without any basic auth happening. ...
0
votes
1
answer
199
views
How to scrape only some targets of EC2 based on tag
I have a requirement, where I have enabled NGINX metrics and I am trying to plot metrics with Prometheus. I am trying to plot metrics for NGINX EC2 server which has a tag named test=nginx or by ...
- 111
0
votes
0
answers
20
views
Can I use statsD to scrape Kubernetes metric endpoint or from Kubernetes metric exporters?
Can I use statsD to scrape Kubernetes metric endpoint or from Kubernetes metric exporters(for kubernetes infrastructure metrics like nodes, pods, control plane)?
I see people are using Prometheus more....
- 101
0
votes
0
answers
56
views
Is there a way to send alert emails if Kubernetes detects an authentication attempt, even if I don't have access to the master node?
I have a Kubernetes cluster in OVH, and I need to send alerts if the API detects an authentication attempt. I know I could use a script to monitor Kubernetes audit log files and send emails if there ...
1
vote
1
answer
72
views
get node_exporter metrics for a single system user
I have installed the node_exporter within the userspace of a linux system user. Everything works and I get the metrics in my grafana. But what I get is apperently the metrics for the whole host system ...
- 25
3
votes
1
answer
201
views
Prometheus and Azure WebApps: How to monitor multiple instances with prometheus
I have an application that has two instances running and each request runs on a different machine.
I am monitoring an event with a counter in my application and each instance runs it's own counter ...
- 131
1
vote
1
answer
1k
views
Prometheus scrape from service vs pods
Let's say I have multiple pods (replicas) of application under a service (with targetPort pointing to port where application keeps its metrics for Prometheus) named "app-monitor-service". ...
deals my
0
votes
2
answers
159
views
Aws EKS Prometheus helm failed pod CrashLoopBackOff with log "trigger reload: reload request failed: Post \"http://127.0.0.1:9090/-/reload\"
I followed the steps here https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-onboard-ingest-metrics-new-Prometheus.html
In a nutshell, I installed the Prometheus community helm chart on my ...
- 101
0
votes
1
answer
529
views
Metrics Server and Prometheus have overwriting resources
I am trying to install Metrics Server, but it needs to create a resource that already exists from an installation of Prometheus. They both want to make v1beta1.metrics.k8s.io.
I went to see if the ...
- 206
3
votes
1
answer
174
views
Measuring image pull latency in k8s cluster
I maintain a Kubernetes clusters that pulls a number of images (mostly from Docker Hub, but also some from GCR and GHCR). I am interested in building a dashboard to measure the amount of time it takes ...
- 135
0
votes
2
answers
901
views
How to defragment etcd database in Kubernetes cluster
I am trying to resolve a recurring Prometheus alert I'm getting that states that my Kubernetes cluster etcd database is fragmented and should be defragmented. I have never used the etcdctl command ...
- 103
0
votes
1
answer
263
views
List all active promtail targets
I have spend a significant time reading the manual, but I still cannot figure out how to actually get a list of the targets that promtail is currently tailing.
The metrics endpoint only exposes ...
0
votes
1
answer
2k
views
Add more static instance labels in prometheus config
I'm adding serves to my prometheus config like this:
- job_name: cluster1
static_configs:
- targets:
- server1.example.com
- server2.example.com
- server3.example....
- 25
-1
votes
1
answer
40
views
Enterprise Ready Prometheus
I am new to Prometheus and have been reading about its popularity as a tool for monitoring applications, containers and K8s objects. However, most of the blogs that I have come across have at best ...
- 187
1
vote
1
answer
556
views
Prometheus scrape config for kubelet node role not auth'ing
I have a kubernetes (not minikube) cluster (1.25) installed on a dedicated machine (1-node cluster). I installed Prometheus. I am using a ServiceAccount, Role, ClusterRoleBinding like this:
apiVersion:...
- 111
1
vote
1
answer
811
views
How to add an aditional ServiceMonitor for prometheus-operator?
I installed prometheus-operator using kube-prometheus-stack and so far everything has worked fine, but I need to add one more service to monitoring. This is a Windows Server that is not running on the ...
- 13
2
votes
0
answers
48
views
Why is an 'average' as a metric favored (over 'latency') in this case?
In a talk on RED method, the presenter defends his use of 'average' by the following comparison with latencies. The linked-to part of the talk written down (non-verbatim and with my formatting):
...
- 121
3
votes
0
answers
204
views
How to collect statistics from CI over time?
CI environments like GitLab (self-hostet and Cloud), GitHub, CodeCov, Codacy, ... collect statistics over time, so a developer or team lead can see the evolution of the project over time:
number of ...
- 131
0
votes
1
answer
937
views
Kubernetes Job Metrics in Prometheus
I want to create a Kubernetes Job object like the example: https://kubernetes.io/docs/concepts/workloads/controllers/job/
Now imagine I want to have the result (in this case the value of Pi: 3.14159......
0
votes
1
answer
907
views
How to purge / prevent using the prometheus local storage when using remote_write
I configured Prometheus to push metrics using remote_write but the local storage is still in use and keeps growing
How could Prometheus be configured to stop storing locally and only push metrics or ...
- 101
0
votes
1
answer
277
views
Snapshot or Saving Prometheus Data for Performance testing and comparison
I have an application and system that exposes its resource metrics to Prometheus and Grafana that I want to do performance testing on. I want to be able to conduct a test, capture the metrics on CPU, ...
- 191
3
votes
1
answer
4k
views
What is the difference between Fluent Bit and Prometheus?
From what I've seen, Prometheus scrapes data and metrics and displays them and Fluent Bit collects logs and metrics and can output them.
So, are they doing the same thing, i.e. should I choose one ...
- 33
0
votes
1
answer
94
views
Apache Camel-k metrics in rancher embedded prometheus
I have a Kubernetes running Rancher 2.4.3. I have the cluster monitoring enabled in rancher, so that exists a Prometheus instance running, so as a Prometheus Operator.
Recently I deployed a Apache ...
- 101
2
votes
1
answer
3k
views
Does Prometheus log scrape-duration?
I'd like to graph scrape duration for a particular target.
The Prometheus web interface has a Status of Targets page, which tabulates the (time elapsed since) "last scrape" and corresponding ...
- 289
0
votes
1
answer
781
views
Scraping different Prometheus metrics at different rates
I am running a Tendermint network. I am using Prometheus to track various metrics, and to track node health. For performance metrics such as tendermint_consensus_total_txs, I want updates every second....
- 103
3
votes
0
answers
1k
views
Create a Kubernetes Service that loads an external host url AND port
I have a url like this: http://myserver.mydomain.com:98765
I need to have this url be a Kubernetes service so I can address it in Kubernetes like this: my-service.mynamespace.svc.cluster.local. (So, ...
- 206
2
votes
2
answers
4k
views
Monitoring Kubernetes CronJob & Job in Grafana for today day only
I'm trying to create a Dashboard in Grafana that will show the status of Cronjob & Job success & fail.
Currently i'm using kube_job_status_succeeded to get a status job, but the issue here ...
- 21
2
votes
1
answer
210
views
How prometheus scrapes individual pods actuator endpoints when they are exposed via a single service in kubernetes?
I have created a deployment with 2 replica sets. But the Service exposes the 8080 and the management endpoint using NodePort. How should I set up the pods deployment if Prometheus deployment should ...
- 133
0
votes
1
answer
778
views
How to setup basic auth for Prometheus deployed on K8s cluster using yamls?
How to setup basic auth for Prometheus deployed on K8s cluster using yamls??
I was able to achieve this easily when Prometheus was deployed on a host locally using tar file. But when it is deployed as ...
- 1
1
vote
2
answers
5k
views
Enable ingress-nginx Prometheus monitoring without Helm
A few weeks ago, I installed ingress-nginx in my Kubernetes cluster. Now, I realize that was a mistake because that did not come with Prometheus monitoring enabled automatically, and even worse, it ...
- 115
2
votes
0
answers
1k
views
Unable to fetch metrics from custom metrics API: Internal error occurred
I am running a local Kubernetes cluster with Minikube on a Windows machine using --driver=hyperv.
I am trying to set up an HPA (HorizontalPodAutoscaler) based on Pods custom metrics. Using Helm, I ...
- 21
0
votes
2
answers
930
views
Getting error while trying to communicate with apiserver in kube-state-metrics pod
I am getting error:
# kubectl logs kube-state-metrics-fdc49658-g98sl -n monitoring
I0523 11:30:25.206556 1 main.go:86] Using default collectors
I0523 11:30:25.206766 1 main.go:98] Using ...
- 151
1
vote
1
answer
545
views
Where to install Prometheus for Kubernetes monitoring
I'm considering using Prometheus to monitor and analyze different metrics from my AWS EKS Kubernetes cluster running in Fargate, as it seems to be the standard tool during these times. However, ...
2
votes
1
answer
4k
views
How to filter label_values in Loki?
I want to create a query variable in grafana using Loki that returns all the possible label values of the label baz such that they only include label values that exist where the label foo equals "...
- 151