Questions tagged [prometheus]
For questions about Prometheus, an open-source systems monitoring and alerting toolkit.
72
questions
0
votes
0
answers
18
views
Prometheus HTTP API returns no metrics data for valid target
I have a target in Prometheus that's functional and is polling successfully; Grafana is able to see the data from Prometheus and render appropriate charts/dashboards.
However, when I poll the HTTP API ...
- 101
0
votes
0
answers
12
views
Add more static instance labels in prometheus config
I'm adding serves to my prometheus config like this:
- job_name: cluster1
static_configs:
- targets:
- server1.example.com
- server2.example.com
- server3.example....
- 1
0
votes
1
answer
23
views
Enterprise Ready Prometheus
I am new to Prometheus and have been reading about its popularity as a tool for monitoring applications, containers and K8s objects. However, most of the blogs that I have come across have at best ...
- 189
0
votes
0
answers
20
views
How to calculate Jenkins Job increase in build time through Prometheus query
I extract statistics from Jenkins with the Prometheus Metrics plugin.
I have created a query in PromQL to check if a Jenkins job builds time has increased by 50% from the average successful build time:...
1
vote
1
answer
70
views
Prometheus scrape config for kubelet node role not auth'ing
I have a kubernetes (not minikube) cluster (1.25) installed on a dedicated machine (1-node cluster). I installed Prometheus. I am using a ServiceAccount, Role, ClusterRoleBinding like this:
apiVersion:...
- 111
0
votes
0
answers
74
views
How to add an aditional ServiceMonitor for prometheus-operator?
I installed prometheus-operator using kube-prometheus-stack and so far everything has worked fine, but I need to add one more service to monitoring. This is a Windows Server that is not running on the ...
- 1
2
votes
0
answers
19
views
Why is an 'average' as a metric favored (over 'latency') in this case?
In a talk on RED method, the presenter defends his use of 'average' by the following comparison with latencies. The linked-to part of the talk written down (non-verbatim and with my formatting):
...
0
votes
0
answers
18
views
Access GCP Managed Prometheus metrics from Grafana on Windows
I have installed Grafana (running at localhost:3000) and Prometheus (running at localhost:9090) on Windows 10, and am able to add the latter as a valid data source to the former. However, I want to ...
1
vote
0
answers
126
views
Can't access default k3s metrics server from Prometheus-Adapter
I have installed Prometheus-adapter along with the default metrics-server that comes with k3s securely on port 4443.
Unfortunately, I get no resources when I query custom.metrics.k8s.io
$ kubectl get ...
- 111
1
vote
0
answers
65
views
How to collect statistics from CI over time?
CI environments like GitLab (self-hostet and Cloud), GitHub, CodeCov, Codacy, ... collect statistics over time, so a developer or team lead can see the evolution of the project over time:
number of ...
- 111
0
votes
0
answers
9
views
On Demand Trigger Scape Prometheus
I'm pushing metrics to a push gateway. Imagine my scrape_interval is 5mins. I need to move on with the process. I don't want to wait 5 mins for data.
I'd like to trigger a scrape of the push gateway ...
0
votes
1
answer
239
views
Kubernetes Job Metrics in Prometheus
I want to create a Kubernetes Job object like the example: https://kubernetes.io/docs/concepts/workloads/controllers/job/
Now imagine I want to have the result (in this case the value of Pi: 3.14159......
0
votes
0
answers
92
views
Loki-stack helm chart is not allowing to edit grafana.ini
I have deployed Loki-stack using helm charts. For configuring email alerts, I need to modify grafana.ini. I am not able to edit grafana.ini as it has read-only permissions.
Even I could find a way to ...
0
votes
0
answers
559
views
basic authentication prometheus pod
I'm trying to add basic authentication for prometheus pod but when I deploy the pod with the below config I get the error: "Readiness probe failed: HTTP probe failed with statuscode: 401", I ...
0
votes
0
answers
71
views
Prometheus not discovering Kubernetes service based on type and port
I have a Kubernetes cluster that e.g. hosts service/my-service with the following key characteristics:
apiVersion: v1
kind: Service
metadata:
name: my-service
namespace: my-namespace
spec:
ports:...
- 141
0
votes
1
answer
287
views
How to purge / prevent using the prometheus local storage when using remote_write
I configured Prometheus to push metrics using remote_write but the local storage is still in use and keeps growing
How could Prometheus be configured to stop storing locally and only push metrics or ...
- 101
0
votes
1
answer
131
views
Snapshot or Saving Prometheus Data for Performance testing and comparison
I have an application and system that exposes its resource metrics to Prometheus and Grafana that I want to do performance testing on. I want to be able to conduct a test, capture the metrics on CPU, ...
- 181
0
votes
0
answers
1k
views
How do I delete ALL time series in prometheus without restarting?
How do I delete all data?
I tried:
Using the HTTP API.
Deleting a series with a match operator such as
curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={job="BR&...
- 101
1
vote
1
answer
1k
views
What is the difference between Fluent Bit and Prometheus?
From what I've seen, Prometheus scrapes data and metrics and displays them and Fluent Bit collects logs and metrics and can output them.
So, are they doing the same thing, i.e. should I choose one ...
- 13
0
votes
1
answer
68
views
Apache Camel-k metrics in rancher embedded prometheus
I have a Kubernetes running Rancher 2.4.3. I have the cluster monitoring enabled in rancher, so that exists a Prometheus instance running, so as a Prometheus Operator.
Recently I deployed a Apache ...
- 101
2
votes
1
answer
858
views
Does Prometheus log scrape-duration?
I'd like to graph scrape duration for a particular target.
The Prometheus web interface has a Status of Targets page, which tabulates the (time elapsed since) "last scrape" and corresponding ...
- 205
0
votes
1
answer
208
views
Scraping different Prometheus metrics at different rates
I am running a Tendermint network. I am using Prometheus to track various metrics, and to track node health. For performance metrics such as tendermint_consensus_total_txs, I want updates every second....
- 103
3
votes
0
answers
469
views
Create a Kubernetes Service that loads an external host url AND port
I have a url like this: http://myserver.mydomain.com:98765
I need to have this url be a Kubernetes service so I can address it in Kubernetes like this: my-service.mynamespace.svc.cluster.local. (So, ...
- 194
2
votes
2
answers
2k
views
Monitoring Kubernetes CronJob & Job in Grafana for today day only
I'm trying to create a Dashboard in Grafana that will show the status of Cronjob & Job success & fail.
Currently i'm using kube_job_status_succeeded to get a status job, but the issue here ...
- 21
2
votes
1
answer
80
views
How prometheus scrapes individual pods actuator endpoints when they are exposed via a single service in kubernetes?
I have created a deployment with 2 replica sets. But the Service exposes the 8080 and the management endpoint using NodePort. How should I set up the pods deployment if Prometheus deployment should ...
- 133
0
votes
1
answer
295
views
How to setup basic auth for Prometheus deployed on K8s cluster using yamls?
How to setup basic auth for Prometheus deployed on K8s cluster using yamls??
I was able to achieve this easily when Prometheus was deployed on a host locally using tar file. But when it is deployed as ...
- 1
1
vote
2
answers
3k
views
Enable ingress-nginx Prometheus monitoring without Helm
A few weeks ago, I installed ingress-nginx in my Kubernetes cluster. Now, I realize that was a mistake because that did not come with Prometheus monitoring enabled automatically, and even worse, it ...
- 115
1
vote
0
answers
684
views
Unable to fetch metrics from custom metrics API: Internal error occurred
I am running a local Kubernetes cluster with Minikube on a Windows machine using --driver=hyperv.
I am trying to set up an HPA (HorizontalPodAutoscaler) based on Pods custom metrics. Using Helm, I ...
- 11
0
votes
1
answer
543
views
Getting error while trying to communicate with apiserver in kube-state-metrics pod
I am getting error:
# kubectl logs kube-state-metrics-fdc49658-g98sl -n monitoring
I0523 11:30:25.206556 1 main.go:86] Using default collectors
I0523 11:30:25.206766 1 main.go:98] Using ...
- 153
1
vote
1
answer
230
views
Where to install Prometheus for Kubernetes monitoring
I'm considering using Prometheus to monitor and analyze different metrics from my AWS EKS Kubernetes cluster running in Fargate, as it seems to be the standard tool during these times. However, ...
2
votes
1
answer
2k
views
How to filter label_values in Loki?
I want to create a query variable in grafana using Loki that returns all the possible label values of the label baz such that they only include label values that exist where the label foo equals "...
- 151
0
votes
1
answer
212
views
Node-centric network metrics are very different from the sum of metrics of the Pods running on that node
Context
I am trying to chart the network bandwidth usage of a node in 2 different manners:
By looking at global metrics for that node
By summing up the corresponding metric for each Pod
To achieve ...
- 111
0
votes
1
answer
130
views
micrometer exposing actuator metrics vs kube-state-metrics vs metrics-server to set pod request/limits
micrometer exposing actuator metrics to set request/limit to pods in K8svs metrics-server vs kube-state-metrics -> K8s Mixin from kube-promethteus-stack Grafana dashboad
It's really blurry and ...
1
vote
2
answers
250
views
Is it possible/fine to run Prometheus, Loki, Grafana outside of Kubernetes?
In some project there are scaling and orchestration implemented using technologies of a local cloud provider, with no Docker & Kubernetes. But the project has poor logging and monitoring, I'd like ...
- 181
0
votes
1
answer
95
views
Grafana Status Panel Plugin - genereate status box for each entry from a Prometheus querry
I am trying to create a dashboad out of process_uptime_seconds{namespace="$namespace"} querry from Spring Actuator. and Prometheus as datasource I am trying to build a single stat box menus ...
3
votes
0
answers
123
views
Grafana K8s microservices availability dashboard based on actuator uptime
I need to create a dashboard that is using micrometer prometheus spring actuator endpoint to show all services for a given namespaces in green boxes or red boxes depending on status, the boxes should ...
2
votes
1
answer
3k
views
Spring boot prometheus - difference between system_cpu_usage and process_cpu_usage
Spring boot reports two statistics when Prometheus in place.
# HELP system_cpu_usage The "recent cpu usage" for the whole system
# TYPE system_cpu_usage gauge
system_cpu_usage 0....
- 243
1
vote
1
answer
108
views
How to append namespace before metric name in prometheus?
I have static config in prometheus.yml
- job_name: 'myjob'
static_configs:
- targets: ['monitoring.api.endpoint.net']
And I recieve various metrics for example cpu_usage, disk_write_ops, ...
1
vote
1
answer
258
views
Is there a way to make Prometheus to scrape only the metrics which are present in Grafana?
As a customer of Grafna Cloud, I need to minimize the number of DataSeries that are being sent (remote_write) to Grafana Cloud, to avoid being overcharged for metrics that I don't really use.
This can ...
- 151
7
votes
3
answers
10k
views
Prometheus Alertamanger - how to silence all alerts for a given period during a maintenance?
A work scenario that I can't cover currently is that I want to set a maintenance mode, meaning all alerts received from Prometheus to be ignored, I want to be able to set it through the UI for a given ...
3
votes
1
answer
61
views
Initial Configuration: Grafana and Prometheus inside docker and then AWS
I'm a bit of a newbie with this, but I haven't been able to find an explanation. Here's my issue. I'm trying to get Prometheus and Grafana configured on an AWS Fargate instance. I'm testing inside a ...
- 141
-1
votes
1
answer
312
views
K8s to Prometheus apiserver_client_certificate_expiration_seconds_count metric, why does it count more seconds than 86400 for a single day?
In the 2 pictures below I am comparing the metric from K8s apiserver_client_certificate_expiration_seconds_count{job="apiserver"} for my 3 master nodes on K8s 1.14.3. A single day should have 86400 ...
1
vote
1
answer
168
views
Reuse Alertmanger Slack Integration into Jiralert template
What kind of changes need to be made to a slack template in order to reuse into Jiralert one? Can I utilize the whole body as it is, can somebody post an example of a working resuse, even for the ...
0
votes
1
answer
201
views
Disable creation of labels in Jira?
Following the base instructions I can't create new Jira item through API request due to failing because of trying to create labels. Even if I am using add_group_labels: false it tries to create label ...
5
votes
2
answers
1k
views
Back-filling prometheus (and related system) metrics?
Projects like Thanos, which are based on prometheus storage formats, seem to say that it is impossible to back-fill data in a prometheus based system.
Can anyone explain why and if there are any ...
- 1,430
3
votes
0
answers
441
views
Thanos or Cortex - What handles very large scale (say, hundreds of millions of time series) better?
We're looking at a new metrics solution and are attempting to build it in house. So, my question is:
What can scale more effectively / with less pain; Thanos or Cortex?
I understand the general ...
- 1,430
1
vote
1
answer
63
views
Pometheus-operator - modify Grafana mixin dashboard
Whenever I try to edit the provisioned dashboards I get an error stating that the provisioned dashboard can't be edited. I saw that there is an option to allow changes from the UI, but my questions is ...
2
votes
1
answer
3k
views
Prometheus - join series based on the key part of a label
In 1.16 k8s deprecated the label kubernetes.io/role: worker
I had previously been using as a filter to target graphs to specific node role using this in my grafana variables declaration
label_values(...
- 101
0
votes
1
answer
1k
views
Prometheus-operator - detection of expiring certs failing
I had a situation yesterday where my K8S master api cert has expired and I had Prometheus-operator active in the past weeks, but neither the 24 hours, neither the 7 days alert has triggered. I went to ...
4
votes
3
answers
197
views
What is the Prometheus and Grafana ideal setup?
I am just wondering. If I have many environments monitored via Prometheus, what will be the best configuration?
The security and efficiency are already important for on-premise installations but ...
- 155