Questions tagged [monitoring]

For questions about observing, tracking, and recording activities or data to measure progress towards achieving objectives.

Filter by
Sorted by
Tagged with
0 votes
1 answer
16 views

Concat multiple meta labels on prometheus for relabling

How can I change the value of the label __param_target to resemble how I would interact with a Kubernetes service in-cluster, following the format: http://serviceName.namespace.svc:servicePort/path ? ...
MrZorkiPongi's user avatar
0 votes
0 answers
10 views

Stateful alerts in Grafana

In Grafana, is there any way to make stateful alerts? For example: Make an alert that triggers if there is any significant (say 5σ) deviation from recent historic trends. Make an alert that only ...
benjimin's user avatar
  • 289
0 votes
0 answers
52 views

Promtail does not sent logs after restarting the host

I have a Loki server that receives logs from other servers. The logs from containers have no problem(grafana/loki-docker-driver). However, other logs(Promtail) from log files are not updated after ...
Amir Sabeghi's user avatar
0 votes
0 answers
81 views

Blackbox exporter refuses to verify endpoints which use selfSigned certificates, openshift cluster

We have an openshift cluster with a prometheus operator monitoring stack. Blacbkox exporter is deployed, configured and is running. I want to probe /actuator/health endpoints of Spring Boot ...
Seidz_q's user avatar
0 votes
1 answer
38 views

How to aggregate logs and metrics from multiple modules accross different servers?

We are thinking about setting up a solution for aggregating logs and metrics from multiple modules accross different servers. The problem is that it needs to run in a hospital and the logs may contain ...
ssindelar's user avatar
  • 103
0 votes
0 answers
8 views

Send failure notification only for a scheduled execution

In Rundeck, how can job failure notifications be sent only for a scheduled executions, not for manually triggered ones? As manual executions are normally monitored specifically by the person running ...
Jānis Elmeris's user avatar
0 votes
0 answers
52 views

Why Kube state metrics only shows metrics related to the namespace where it is running?

I have AWS EKS cluster with kube-state-metrics installed in a namespace called "monitoring". This installation is using service monitor and other components (see yaml files below). In this ...
Thiago Scodeler's user avatar
0 votes
0 answers
88 views

Monitor Container Load Average with Grafana Agent and Cadvisor (Docker Integration)

I’ve configured Docker integration on Grafana Cloud, utilizing Cadvisor and Grafana Agent to monitor Docker containers, and it’s been effective for most metrics. I’ve been referencing the metrics ...
Liquid_Soap's user avatar
0 votes
0 answers
171 views

Configure Azure Container Insights so that metrics are not sent to Log Analytics Workspace?

I want to configure my Container Insights monitoring on my AKS cluster so that only Container Insights logs and not metrics are sent to my log analytics workspace. I still want metrics to be collected ...
floaty39's user avatar
1 vote
0 answers
102 views

Can Loki accept/combine a log stream (unique set of labels) from multiple Promtail instances running simultaneously? (applies to Prometheus too)

I am using ec2_sd_configs in Loki (and Prometheus) to discover my EC2 instances, most of which are autoscaled, and therefore come and go. They are identical load balanced API instances, so I don't ...
Life5ign's user avatar
  • 130
0 votes
0 answers
10 views

File integrity monitoring: How to initialise the database of files on an upgrade?

let's go for what I'm not asking first: Product recommendations End-to-end solutions (which will inevitably end up in an argument) What I need is a deployment architecture that solves the problem of ...
David Boshton's user avatar
1 vote
0 answers
36 views

Best way to rollout open telemetry helm operators and resources in a multi-cluster K8s Setup

I am currently tasked with building an observability pipeline based on open telemetry. We are looking at shipping all 3 key signals (metrics, traces, and logs) to the Grafana cloud. We also have a ...
sage's user avatar
  • 199
0 votes
1 answer
31 views

Virtual Machine Monitoring KQL yielding empty results

I have enabled Azure Monitoring in one of our virtual machine. Is been 4 days the virtual machine is on, below is the query: Perf | where ObjectName == "Memory" and (CounterName == "...
SandeshR's user avatar
  • 136
1 vote
1 answer
689 views

What are the good metrics to collect for frontend application?

Context: We have some features that rarely calls any backend service and the logic stays in the frontend. Most of the tools we have can calculate SLI/SLO status based on latency and error code return ...
nvcnvn's user avatar
  • 113
1 vote
1 answer
2k views

Grafana NGINX reverse proxy not functioning correctly

I seem to be having trouble with Grafana behind an nginx proxy. I'm getting console errors and errors in Grafana, and the changes I try to make in Grafana are not working. Errors: From within Grafana ...
baxtmann's user avatar
-1 votes
0 answers
105 views

Restricting one module for one tf file

So I have file called main.tf in which I am calling the existing module for iam_role, but in same main.tf I have called one more module for service_role but I want to restrict that service_role module ...
Mandy's user avatar
  • 1
2 votes
2 answers
3k views

Monitoring Kubernetes CronJob & Job in Grafana for today day only

I'm trying to create a Dashboard in Grafana that will show the status of Cronjob & Job success & fail. Currently i'm using kube_job_status_succeeded to get a status job, but the issue here ...
DagaReiN's user avatar
1 vote
1 answer
1k views

What means "Reconciliation" from Argo CD

I m looking a metric in Grafana from a official Dashboard from Argo CD, the panel its named "Reconciliation Activity" and use the following metric: argocd_app_reconcile_count I look for a ...
Enrique Benito Casado's user avatar
1 vote
0 answers
82 views

How do I monitor response times with sprint actuator metrics and Telegraf?

We are using micrometer with the statsd flavor in our Spring Boot applications to send metrics to Telegraf for visualization in the Grafana dashboard. However, I notice that the most valuable metrics ...
IcedDante's user avatar
  • 111
1 vote
1 answer
928 views

How to monitor ELK using metricbeat on kubernetes (v8 compatible and simple solution)

For probably more than a year there's been a deprecation warning on xpack monitoring in elasticsearch, logstash, kibana etc. saying the built-in monitoring is going to disappear. Since I don't like ...
Kristofer's user avatar
  • 131
2 votes
1 answer
105 views

What is a good strategy to reliably and automatically perform remote system tests of a web service?

In a company I work for, we have deployed many services: a major part of them are small and rather rarely used, but an outage would be a problem nonetheless. At some point I realized that manual ...
d33tah's user avatar
  • 71
1 vote
3 answers
975 views

What is the best way to monitor job status?

I would like to monitor for recent spike in failed application jobs at my company. Job statuses are stored in a MySQL table. We have on order of 50 different types of jobs, and sometimes, one of ...
MedicineMan's user avatar
3 votes
0 answers
90 views

Monitoring SSRS report server with AppDynamics

I've spent several days trying to figure out how to set up AppDynamics to monitor our SQL Server Reporting Servers, and am encountering the issue where multiple nodes for the same server and ...
Stuart Ainsworth's user avatar
2 votes
2 answers
1k views

Log/Data rotation in Docker Container

My question is more on the concept or best practices to follow. I have two docker containers running InfluxDB and Grafana (monitoring applications). I have mounted external volume to each. now, the ...
Chandella07's user avatar
1 vote
1 answer
421 views

Where to install Prometheus for Kubernetes monitoring

I'm considering using Prometheus to monitor and analyze different metrics from my AWS EKS Kubernetes cluster running in Fargate, as it seems to be the standard tool during these times. However, ...
Nicolás García's user avatar
3 votes
0 answers
46 views

Is there an all-in-one control center software that can be used to manage build and test execution and to visualize test results?

If there are no constraints with regards to other components used in the system at all, do you know a software that can be used as a sort of "cockpit" / control center to visualize test ...
Hunter2's user avatar
  • 31
1 vote
1 answer
1k views

Pagerduty API integration and auto-resolution

It is possible to implement auto-resolution with Pagerduty API integration. When a monitoring system detects an issue based on some metric and a threshold, it would create a Pagerduty incident through ...
codeforester's user avatar
0 votes
1 answer
71 views

Simple service to check infrastructure status

I have a very simple setup here (solo IT guy): we have a on-prem server which runs some services. They should produce e.g. a daily logfile. I'd like to get a notification if this doesn't happen for ...
Frank Meulenaar's user avatar
3 votes
1 answer
1k views

Logging & Monitoring on multiple servers

I lead a small project with 2 groups of servers (servers in a single group are clones with the same microservices, like pods in the k8s), a balance loader, and a DB. Docker & k8s aren't used ...
AivanF.'s user avatar
  • 181
1 vote
1 answer
427 views

Is there a way to make Prometheus to scrape only the metrics which are present in Grafana?

As a customer of Grafna Cloud, I need to minimize the number of DataSeries that are being sent (remote_write) to Grafana Cloud, to avoid being overcharged for metrics that I don't really use. This can ...
Meir Gabay's user avatar
3 votes
0 answers
352 views

How to deploy StatsD listening on UDP in kubernetes with external loadbalancer on AWS?

Our goal is to have StatsD deployed on k8s cluster running on AWS. Currently all existing applications using UDP to send StatsD metrics. As we understood from the kubernetes docs and tests we ...
rombob's user avatar
  • 657
1 vote
1 answer
46 views

Monitoring K8s Pods in case of outage

K8s has self-healing builtin features, if a pod crash then it will be replaced with the new pod but its log is gone. So what is the best way to monitor K8s pods in case of outage?
Chau Giang's user avatar
2 votes
1 answer
61 views

Production grade methodology for alerts

Background Our code is written with: Unit tests End to end tests Code review Staging process Deployment process On the contrary, our alerts are just written and then modified occasionally manually. ...
Michael's user avatar
  • 123
4 votes
3 answers
246 views

What is the Prometheus and Grafana ideal setup?

I am just wondering. If I have many environments monitored via Prometheus, what will be the best configuration? The security and efficiency are already important for on-premise installations but ...
gervais.b's user avatar
  • 155
4 votes
1 answer
3k views

monitor for docker stats similar to htop?

I'm in the need for this and my google leaded me to this post on SO I guess it's helpful to boost the beta for our sister site DevOps Stackexchagne here. I quote the original question docker stats ...
Nam G VU's user avatar
  • 195
0 votes
1 answer
3k views

SQS: What could high number of visible messages mean if messages received & deleted are normal?

I'm new to SQS. I recently noticed the ApproximateNumberOfMessagesVisible in my SQS queue started rising. What was interesting though is that there was no sudden surge in the number of messages ...
Shubham Kanodia's user avatar
1 vote
1 answer
45 views

Are there any tools to monitor custom console applications on Windows Server?

Are there any DevOps tools or known practices to monitor custom made console application on an on premises (Windows Server) environment? We developed an exe that needs to be continuously running, ...
carraua's user avatar
  • 113
1 vote
2 answers
192 views

Gather resource usage by process in a kubernetes cluster

I am searching for a tool similar to Prometheus + Grafana that gather and record resource usage especially memory usage by process-ID or process-name. We have two components that are running ...
AVarf's user avatar
  • 205
0 votes
2 answers
70 views

How to define Service Level Objectives without historical data?

Assuming I can define Service Level Indicators for a user journey are there any structured methods for setting a Service Level Objective within that SLA without any historical data to indicate what is ...
Richard Slater's user avatar
0 votes
1 answer
290 views

How to get Docker Container stats using DataDog API?

I have a running DataDog-Agent on a Docker server. Is there a function in the DataDog API to get the stats of every single container, so I could print the stats to an external Webpage? Or is there ...
Deeonix's user avatar
2 votes
2 answers
235 views

How to get Memory Usage from Windows Server 2012 R2 in OpenStack

I have an Rocky OpenStack cluster, I want to monitor memory usage of instances. monitoring of Linux instances is OK but Windows Server instances do not provide any data for memory usage. I've ...
Milad Yarmohammadi's user avatar
6 votes
2 answers
996 views

What is alert fatigue?

Recently, we hear the term "alert fatigue" a lot. I have a few questions regarding this: where did this term come from? was it invented by a company like PagerDuty? what is alert fatigue actually?
codeforester's user avatar
1 vote
1 answer
61 views

What considerations should be taken into account for a good log file format?

We have a practical question I struggle to answer : What is a good logfile schema? Here are the caveats: There are several workloads, each writing more than one log file Applications are on more ...
Bruce Becker's user avatar
  • 3,593
0 votes
1 answer
94 views

Azure monitor and deployment slots

My question is actually quite simple: if I have a webapp with a deployment slot and a monitor that checks the health of webapp in some way (connections, cpu time or the like) , if I do a swap, what is ...
mslot's user avatar
  • 177
1 vote
2 answers
263 views

What is the advantage of using Prometheus over ingesting metrics using logs?

I want to setup monitoring on our environment. I am wondering on what are the advantages of using something like Prometheus for making metrics vs. just using something like the ELK stack for creating ...
J Zap's user avatar
  • 13
0 votes
1 answer
68 views

How to scale the control plane of a service mesh?

Scalability in the data plane is easy to achieve by simply having more replicas. The configuration the proxies receive from the control plane makes that possible . However, if the number of services (...
jhamfler's user avatar
0 votes
1 answer
238 views

Reset time vs alerting window size in alerting strategies for SLOs

In https://landing.google.com/sre/workbook/chapters/alerting-on-slos/#pros_and_cons_of_alerting_based_on_burn_r I don't understand how the reset time can be 58 min when the alerting window size is one ...
user15207's user avatar
3 votes
0 answers
876 views

Kubernetes metric or command to show liveness delay time (app startup time)

HTTP Kubernetes liveness and readiness checks are configured as per the documentation. The initialDelaySeconds field tells the kubelet how long it should wait before performing the first liveness / ...
JamesJJ's user avatar
  • 161
7 votes
1 answer
720 views

How do you monitor and alert on thrashing containers?

Are there any proposed best practices for how to monitor for thrashing containers? We're in a situation where we have containers that might try to come up and will crash -> restart a few dozen times ...
MrDuk's user avatar
  • 295
8 votes
1 answer
6k views

Why is Prometheus not a good choice for data with high cardinality?

I have a background in relational databases and new to Prometheus. I wonder why Prometheus is not a good choice for high cardinality data? Why do I need to use low cardinality data? It's exact ...
Sybil's user avatar
  • 452