14
votes
Accepted
How to create a centralised dashboard to monitor multiple cloud (Amazon and Google)?
I think what you are looking for is an Open Source project that can take inputs from both Amazon CloudWatch and Google StackDriver; there isn't a huge amount out there at the moment, but I will detail ...
- 11.6k
14
votes
Accepted
Prometheus alert CPUThrottlingHigh raised but monitoring does not show it
The CPUThrottlingHigh is an alert created by the kubernetes-mixin project. There is an open issue (#108) to discuss this alert. I suggest that you read all the comments on this issue to better ...
- 935
13
votes
Accepted
Why is Prometheus not a good choice for data with high cardinality?
As far as I know, Prometheus doesn't mind high-cardinality data. What Prometheus doesn't like is high-cardinality labels.
Let's start with Prometheus official documentation, it gives a good high-...
- 1,955
11
votes
Accepted
What exactly is the cloud term 'a Firehose'?
The fire hose is a reference to an old Weird Al Yankovick movie, UHF (1989). In this movie, George Newman (portrayed by Yankovick) starts a local TV channel in the UHF band and is responsible for ...
- 3,724
10
votes
Accepted
How to check Jenkins build executor free memory?
Have a look at the Monitoring plugin. Here are some more details about it (from the linked page):
Charts of memory, cpu, system load average, http response times by day, week, month, year or ...
- 7,185
9
votes
Accepted
Automatic periodic conditional checks
All in all any monitoring tool would do for this case, monit, nagios, shinken, icinga, centreon, or even a crontab in bash could do...
- 11.3k
8
votes
Accepted
Are there any best practices and/or tools for 'nohup' monitoring?
I'm not sure it's a best practice but what I would do is using filebeat to read the log in near real time and push it to ElasticSearch.
AWS provides an ElasticSearch Service which can save you from ...
- 11.3k
8
votes
Accepted
Monitor program progress on multiple servers
Any time you're running ad-hoc long-running commands, you should step back and rethink your process, because that should be automated, including error handling.
Rather than connecting in to the ...
- 2,761
8
votes
Accepted
How fast does Prometheus data grow?
Prometheus use a timeseries databases with vacuum, the documentation gives some maths to plan your disk consumption:
On average, Prometheus uses only around 1-2 bytes per sample. Thus, to
plan ...
- 11.3k
8
votes
Accepted
What is alert fatigue?
The term Alert Fatigue, also known as Alarm Fatigue has been around for decades. It affects many professions, including medical, technical, and construction industries. According to the Alert Fatigue ...
- 664
7
votes
Accepted
Alarms management with Elasticsearch and Kibana
I have no experience with it myself, but Elastalert (http://elastalert.readthedocs.io/en/latest/elastalert.html) sounds like what you need (for the alarms/threshold part)
But besides that, it sounds ...
- 216
7
votes
What is alert fatigue?
I have spend years developing a variety of monitoring tools which employ a variety of approaches to track data points ranging from filtered logs, collecting system stats (i.e.: querying data from /...
- 181
6
votes
Accepted
monitoring/alerting system which can support very arbitrary data source and logic
Consider using something like nagios or Zabbix. While it might be nice to avoid custom queries and alerts, this is impossible. The odds of someone using your specific suite of tools in your ...
- 3,724
5
votes
Monitoring checklist - What things should I be monitoring?
I like this video: GOTO 2016 • Monitoring Microservices • Tom Wilkie
One of the key ideas (for me at least) is to realize the difference between host monitoring and application monitoring. Basically ...
- 200
5
votes
How to check Jenkins build executor free memory?
As for workaround, the actual memory can be checked by invoking Groovy commands directly in Script Console (at /computer/(master)/script). Example command:
println "free -m".execute().text
- 7,781
5
votes
How do I troubleshoot missing data in my Prometheus database?
I think your can do some kind of alerting on a metric rate with something like this:
ALERT DropInMetricsFromExporter
IF rate(<metric_name>[1m]) == 0
FOR 3m
ANNOTATIONS {
summary = "...
- 11.3k
5
votes
Accepted
How to get memory usage per process with sar, sysstat?
As Tensibai mentioned, you can extract this info from the /proc filesystem, but in most cases you need to determine the trending yourself. There are several places which could be of interest:
/proc/[...
- 6,730
5
votes
Laptop Internet connection monitoring
I've started abusing SyncThing for this purpose (among others), configuring it as a system service on laptops, then locking down it's UI to prevent it from being able to be used by a local connection ...
- 1,045
5
votes
Accepted
Sensu TCP Connection Issue
Ends up the rabbitmq logs held the answer. I saw that something was wrong, but the messages were so cryptic I couldn't tell what was actually breaking. Turns out it was an SSL issue caused by an old ...
- 313
5
votes
monitoring/alerting system which can support very arbitrary data source and logic
Adding my brick to the list here, I'm at a point where updating continuously the monitoring system to add new instances and cleanup old ones is a pain.
As such I've turn toward prometeus which works ...
- 11.3k
5
votes
Accepted
Monitoring immutable infrastructure virtual machines
I've looked a bit at Consul and Prometheus and am thinking that that is the way forward?
In short, yes.
Consul is designed exactly for this - it will run health checks against all of your services ...
- 196
5
votes
Health monitoring of web pages without using GET check
I see two way to solve this problem:
Check your logs often and ensure there was an access with code 200 in the last N seconds/minutes and no code 5xx meaning there's a server side error. (the often ...
- 11.3k
4
votes
What tools do you use to monitor applications and business-specific data? (apart from Jenkins)
We use Datadog, with metrics pushed from important operational processes using the statsd protocol ( http://docs.datadoghq.com/guides/metrics/ ), so we aren't too deeply tied to Datadog on the ...
- 161
4
votes
Accepted
How do you monitor Node Micro services running inside of Docker Containers?
This is an area where Kubernetes has the correct model, there should be a load balancer between all systems which should have functional health checks.
Once you start to add Nagios, Zabbix or other ...
- 366
4
votes
How do you monitor Node Micro services running inside of Docker Containers?
One commonly used solution(not container-specific) is to build a health check API within your service which tests all the functionality you want monitored (say availability of DBs and other ...
- 334
4
votes
Accepted
What opensource solutions can be used to implement anomaly detection based on Riemann metrics?
I can think of 3 possibility (Not exactly Riemann metrics based, but around anomaly detection):
Outdated but the most efficient in anomaly detectionI know: Shyline + Oculus from Etsy Both tools ...
- 11.3k
4
votes
How to get memory usage per process with sar, sysstat?
Is it a hard requirement to use only sar and sysstat? If not you might want to look at collectl or collectd. These will enable you to study memory usage over time on a per-process granularity. It's ...
- 1,086
4
votes
How to change PagerDuty incident name?
Within the CloudWatch integration in PagerDuty, you can change the "Derive name from" field to something like "Alarm Description".
4
votes
Accepted
Is Serf an added value when a container orchestration platform like kubernetes is used?
Serf is gossip protocol implementation and has absolutely nothing to do with orchestration tools like k8s. Even if such protocol is used internally, then in most cases it isn't available for you ...
- 456
4
votes
Monitor program progress on multiple servers
Graylog
Since two people already advised you to rethink your current process (which I second since it will cause you sleepless nights at some point ;)), I will go another route and recommend a ...
- 140
Only top scored, non community-wiki answers of a minimum length are eligible