Questions tagged [monitoring]
For questions about observing, tracking, and recording activities or data to measure progress towards achieving objectives.
72
questions
1
vote
1answer
16 views
Production grade methodology for alerts
Background
Our code is written with:
Unit tests
End to end tests
Code review
Staging process
Deployment process
On the contrary, our alerts are just written and then modified occasionally manually. ...
0
votes
0answers
27 views
In K8s - Prometheus, which endpoint to scrape from to get container_cpu_cfs_throttled_seconds_total metric
Scenario
I have an Amazon K8s cluster (EKS) and am starting out with Prometheus & K8s monitoring.
It seems to me there are so many endpoints to scrape from. Here are what I have applied:
node-...
4
votes
2answers
85 views
What is the Prometheus and Grafana ideal setup?
I am just wondering. If I have many environments monitored via Prometheus, what will be the best configuration?
The security and efficiency are already important for on-premise installations but ...
1
vote
1answer
57 views
monitor for docker stats similar to htop?
I'm in the need for this and my google leaded me to this post on SO
I guess it's helpful to boost the beta for our sister site DevOps Stackexchagne here.
I quote the original question
docker stats ...
1
vote
0answers
41 views
How to get alert for LDAP issues
I want to run some script which checks for LDAP issue, and in case it found something, it sent me alert.
I thought using ssh command with some generic user, but I'm not sure what is the response I'm ...
0
votes
1answer
105 views
SQS: What could high number of visible messages mean if messages received & deleted are normal?
I'm new to SQS.
I recently noticed the ApproximateNumberOfMessagesVisible in my SQS queue started rising. What was interesting though is that there was no sudden surge in the number of messages ...
1
vote
1answer
21 views
Are there any tools to monitor custom console applications on Windows Server?
Are there any DevOps tools or known practices to monitor custom made console application on an on premises (Windows Server) environment?
We developed an exe that needs to be continuously running, ...
1
vote
0answers
20 views
Alerting via Slack. Automatically close when “reverse” alert found
Let me explain with an example (contrived). Say I have a script that alerts via slack "Job foo1 started on serverA" and if it dies alerts "Job foo1 ended on serverA". Is there an existing or related ...
0
votes
0answers
22 views
How much is DevOps monitoring following SCADA - Supervisory Control and Data Acquisition?
While we setup monitoring for various parts of our DevOps toolchains (not just operations of deployed assets!), does this mean we build SCADA and might apply its principles?
Or are there some ...
1
vote
2answers
30 views
Gather resource usage by process in a kubernetes cluster
I am searching for a tool similar to Prometheus + Grafana that gather and record resource usage especially memory usage by process-ID or process-name.
We have two components that are running ...
0
votes
2answers
23 views
How to define Service Level Objectives without historical data?
Assuming I can define Service Level Indicators for a user journey are there any structured methods for setting a Service Level Objective within that SLA without any historical data to indicate what is ...
0
votes
1answer
36 views
How to get Docker Container stats using DataDog API?
I have a running DataDog-Agent on a Docker server.
Is there a function in the DataDog API to get the stats of every single container, so I could print the stats to an external Webpage?
Or is there ...
2
votes
2answers
109 views
How to get Memory Usage from Windows Server 2012 R2 in OpenStack
I have an Rocky OpenStack cluster, I want to monitor memory usage of instances.
monitoring of Linux instances is OK but Windows Server instances do not provide any data for memory usage.
I've ...
4
votes
2answers
390 views
What is alert fatigue?
Recently, we hear the term "alert fatigue" a lot. I have a few questions regarding this:
where did this term come from?
was it invented by a company like PagerDuty?
what is alert fatigue actually?
1
vote
1answer
32 views
What considerations should be taken into account for a good log file format?
We have a practical question I struggle to answer :
What is a good logfile schema?
Here are the caveats:
There are several workloads, each writing more than one log file
Applications are on more ...
0
votes
1answer
35 views
Azure monitor and deployment slots
My question is actually quite simple: if I have a webapp with a deployment slot and a monitor that checks the health of webapp in some way (connections, cpu time or the like) , if I do a swap, what is ...
1
vote
2answers
72 views
What is the advantage of using Prometheus over ingesting metrics using logs?
I want to setup monitoring on our environment. I am wondering on what are the advantages of using something like Prometheus for making metrics vs. just using something like the ELK stack for creating ...
0
votes
1answer
32 views
How to scale the control plane of a service mesh?
Scalability in the data plane is easy to achieve by simply having more replicas. The configuration the proxies receive from the control plane makes that possible . However, if the number of services (...
0
votes
1answer
79 views
Reset time vs alerting window size in alerting strategies for SLOs
In https://landing.google.com/sre/workbook/chapters/alerting-on-slos/#pros_and_cons_of_alerting_based_on_burn_r I don't understand how the reset time can be 58 min when the alerting window size is one ...
1
vote
0answers
246 views
Kubernetes metric or command to show liveness delay time (app startup time)
HTTP Kubernetes liveness and readiness checks are configured as per the documentation.
The initialDelaySeconds field tells the kubelet how long it should wait before performing the first liveness / ...
6
votes
1answer
62 views
How do you monitor and alert on thrashing containers?
Are there any proposed best practices for how to monitor for thrashing containers? We're in a situation where we have containers that might try to come up and will crash -> restart a few dozen times ...
7
votes
1answer
2k views
Why is Prometheus not a good choice for data with high cardinality?
I have a background in relational databases and new to Prometheus. I wonder why Prometheus is not a good choice for high cardinality data? Why do I need to use low cardinality data? It's exact ...
1
vote
1answer
100 views
What are the levels of monitoring and their importance?
We are going through a process of change in our IT Operations Management. One of the items we are trying to improve is our monitoring of our infrastructure.
So far I've been thinking of only 3 levels ...
2
votes
0answers
16 views
What Monitoring Dashboards do exist with customizable actions like handles, levers and actionable buttons? [closed]
Are there DevOps web dashboards (open-source preferably) or dashboard frameworks that allow setting up custom buttons, levers which execute custom actions from browser?
Think of "Plus" button which ...
4
votes
1answer
3k views
Prometheus alert CPUThrottlingHigh raised but monitoring does not show it
I have installed Prometheus to monitor my installation and it is frequently raising alerts about CPU throttling.
The Prometheus alert rules to identify this alert is :
alert: CPUThrottlingHigh
expr: ...
5
votes
4answers
346 views
How do you monitor status of multiple docker containers?
My home server has about 15 distinct services running, each represented by a Docker container. Depending on how the developers implemented the Dockerfile, some services have health checks, but most do ...
0
votes
1answer
113 views
Specifying branch name for jenkins job through AnyStatus
I am wondering if anyone else has integrated before a parameterized Jenkins build with AnyStatus.
When removing the parameter above, I am able to run builds using the branch specified previously from ...
2
votes
1answer
36 views
Is there any way to get New Relic or PagerDuty to not alert during nightly maintenance?
Initial Problem
We have some badly behaved overnight maintenance jobs that cause alerts for AppDex and error rates for our apps. These are metrics that would be problematic if they happened during ...
3
votes
1answer
62 views
How to monitor a micro-service oriented solution?
I run a micro-service oriented system, running under a private network, composed mainly of:
databases (clusters)
internal API's
external/public API's
workers (services)
monitoring servers
All are ...
4
votes
1answer
1k views
How fast does Prometheus data grow?
This may sound like a vague question so I will provide some context below. The basic question is :
What parameters describe the growth in size of a Prometheus database over time?
To tie the question ...
2
votes
1answer
45 views
How to monitor a REST call response?
My project is using customer.io to send emails in different flows.
Every now and then we have a bug, and one of those calls stops and email is not sent.
I want to get alerted when an email is not ...
2
votes
1answer
75 views
What do you tell someone new to going oncall or PagerDutyl?
[[ Based on tags and the wording this sounds like a PagerDuty-specific question, but it could apply to any tech or monitoring stack in my mind.... PagerDuty is just so dominant in SF-based jobs I've ...
5
votes
3answers
772 views
How to monitor AWS S3 cross-region replication process?
Amazon S3 has a cross-region replication which will handle copy of new/updated objects to additional region.
The problem is that solution does not provide visibility on state for replication process, ...
3
votes
1answer
254 views
tools to gather variant information from thousands of servers and update into database
Recently, I use ansible to gather information periodically from thousands of servers(which already installed collectd to gather CPU, memory, disks, netowrking info) in our company.
For example: I ...
0
votes
2answers
190 views
How to monitor ELK Stack Load?
We have tried Prometheus and Grafana. Currently it is not set up on our cluster and we are tight with resources.
What are the best ways to monitor load on an ELK stack?
Is there anything built in?
0
votes
3answers
2k views
System monitoring and reporting with Jenkins
Are there any plugins that would enable me to distribute system monitoring and reporting via Jenkins, if I provided it with data in a required format. E.g. I would like to write a simple script that ...
3
votes
5answers
82 views
Health monitoring of web pages without using GET check
I am looking for a solution to monitor the health (DOWN or UP) of my web pages.
Let's say I have three services,
site.com/data
site.com/user
site.com/id
Now, what I can do is that I can create a GET ...
4
votes
1answer
88 views
Infrastructure Stability Report
We are using Azure IAAS to host our applications, and these are hosted on a number of virtual machines for different tiers
Our client have recently asked us to provide monthly report on the ...
2
votes
1answer
896 views
Open source microservice visualization
We run various microservices on rancher and kubernetes, and I am wondering if there are any open source projects that assist in visualizing network traffic between services to help detect dependencies....
1
vote
0answers
40 views
How can we monitor internal or external webservices? [closed]
We have Web Performance monitoring third party Tool. We are using New Relic.
However, we are not able to monitor either Internal or External Web services.
In an assumption, if the web service goes ...
1
vote
1answer
541 views
How effective is ELK than other Monitoring tools?
"ELK" is the acronym for three open source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine. Logstash is a server‑side data processing pipeline that ...
1
vote
2answers
76 views
Monitoring tool to microservices on Kubernetes
I'm deploying an instance with kubernetes on Azure public cloud, and I need a tool to monitor the microservices and pods inside this instance, the question is, what tool are you using to do that?
I've ...
1
vote
0answers
35 views
What does applying maxSeries to a single metric mean? [closed]
As per the graphite documentation for maxSeries,
Takes one metric or a wildcard seriesList. For each datapoint from
each metric passed in, pick the maximum value and graph it.
For multiple ...
2
votes
1answer
163 views
Monitoring immutable infrastructure virtual machines
Currently I am using the HashiCorp stack (Packer, Terraform) and Ansible, to generate Immutable Infrastructure and Infrastructure as Code.
This builds a Virtual Machine from a base image, provisions ...
1
vote
1answer
54 views
Application monitoring in multiple EC2 instances
I am running 10 AWS EC2 instances. Each of them running different applications. I am looking for application level monitoring solution. Suppose, if Apache fails in any instance, it should notify me ...
2
votes
3answers
163 views
Automatic periodic conditional checks
I need a simple tool that will carry out some conditional checks periodically and give me a red/green status that I can use as an indication that something is wrong.
These are examples of checks that ...
9
votes
4answers
150 views
Monitor program progress on multiple servers
We have three servers that are running python programs that are running data analysis tasks inside a tmux session. The method we are using at the moment is ssh'ing into each of them connecting the ...
5
votes
2answers
149 views
Trigger PagerDuty alert after a number of incidents
Is it possible to only have an alert triggered after a certain number of incidents from an integration?
For example, if my application reports a single non-critical failure of a kind, that's worth-...
0
votes
1answer
229 views
OMD vs Ubuntu install issues
When installing OMD 1.3 on Ubuntu, I get the following output:
dpkg: dependency problems prevent configuration of omd-1.31.20160708:
omd-1.31.20160708 depends on libapache2-mod-proxy-html; however: ...
5
votes
4answers
534 views
monitoring/alerting system which can support very arbitrary data source and logic
I am looking for a monitoring/alerting tool/system which can support very arbitrary data source and logic.
(following is a made up example to more easily demonstrate my needs)
I have a database ...
