14 votes
Accepted

How to create a centralised dashboard to monitor multiple cloud (Amazon and Google)?

I think what you are looking for is an Open Source project that can take inputs from both Amazon CloudWatch and Google StackDriver; there isn't a huge amount out there at the moment, but I will detail ...
user avatar
13 votes
Accepted

Prometheus alert CPUThrottlingHigh raised but monitoring does not show it

The CPUThrottlingHigh is an alert created by the kubernetes-mixin project. There is an open issue (#108) to discuss this alert. I suggest that you read all the comments on this issue to better ...
user avatar
13 votes
Accepted

Why is Prometheus not a good choice for data with high cardinality?

As far as I know, Prometheus doesn't mind high-cardinality data. What Prometheus doesn't like is high-cardinality labels. Let's start with Prometheus official documentation, it gives a good high-...
user avatar
  • 1,925
11 votes
Accepted

What exactly is the cloud term 'a Firehose'?

The fire hose is a reference to an old Weird Al Yankovick movie, UHF (1989). In this movie, George Newman (portrayed by Yankovick) starts a local TV channel in the UHF band and is responsible for ...
user avatar
  • 3,655
10 votes
Accepted

How to check Jenkins build executor free memory?

Have a look at the Monitoring plugin. Here are some more details about it (from the linked page): Charts of memory, cpu, system load average, http response times by day, week, month, year or ...
user avatar
  • 6,848
9 votes
Accepted

Automatic periodic conditional checks

All in all any monitoring tool would do for this case, monit, nagios, shinken, icinga, centreon, or even a crontab in bash could do...
user avatar
  • 11k
8 votes
Accepted

Are there any best practices and/or tools for 'nohup' monitoring?

I'm not sure it's a best practice but what I would do is using filebeat to read the log in near real time and push it to ElasticSearch. AWS provides an ElasticSearch Service which can save you from ...
user avatar
  • 11k
8 votes
Accepted

Monitor program progress on multiple servers

Any time you're running ad-hoc long-running commands, you should step back and rethink your process, because that should be automated, including error handling. Rather than connecting in to the ...
user avatar
8 votes
Accepted

How fast does Prometheus data grow?

Prometheus use a timeseries databases with vacuum, the documentation gives some maths to plan your disk consumption: On average, Prometheus uses only around 1-2 bytes per sample. Thus, to plan ...
user avatar
  • 11k
8 votes
Accepted

What is alert fatigue?

The term Alert Fatigue, also known as Alarm Fatigue has been around for decades. It affects many professions, including medical, technical, and construction industries. According to the Alert Fatigue ...
user avatar
  • 580
7 votes
Accepted

Alarms management with Elasticsearch and Kibana

I have no experience with it myself, but Elastalert (http://elastalert.readthedocs.io/en/latest/elastalert.html) sounds like what you need (for the alarms/threshold part) But besides that, it sounds ...
user avatar
  • 216
7 votes

What is alert fatigue?

I have spend years developing a variety of monitoring tools which employ a variety of approaches to track data points ranging from filtered logs, collecting system stats (i.e.: querying data from /...
user avatar
6 votes
Accepted

monitoring/alerting system which can support very arbitrary data source and logic

Consider using something like nagios or Zabbix. While it might be nice to avoid custom queries and alerts, this is impossible. The odds of someone using your specific suite of tools in your ...
user avatar
  • 3,655
5 votes

Monitoring checklist - What things should I be monitoring?

I like this video: GOTO 2016 • Monitoring Microservices • Tom Wilkie One of the key ideas (for me at least) is to realize the difference between host monitoring and application monitoring. Basically ...
user avatar
  • 190
5 votes

How to check Jenkins build executor free memory?

As for workaround, the actual memory can be checked by invoking Groovy commands directly in Script Console (at /computer/(master)/script). Example command: println "free -m".execute().text
user avatar
  • 7,011
5 votes

How do I troubleshoot missing data in my Prometheus database?

I think your can do some kind of alerting on a metric rate with something like this: ALERT DropInMetricsFromExporter IF rate(<metric_name>[1m]) == 0 FOR 3m ANNOTATIONS { summary = "...
user avatar
  • 11k
5 votes
Accepted

How to get memory usage per process with sar, sysstat?

As Tensibai mentioned, you can extract this info from the /proc filesystem, but in most cases you need to determine the trending yourself. There are several places which could be of interest: /proc/[...
user avatar
5 votes

Laptop Internet connection monitoring

I've started abusing SyncThing for this purpose (among others), configuring it as a system service on laptops, then locking down it's UI to prevent it from being able to be used by a local connection ...
user avatar
  • 1,005
5 votes
Accepted

Sensu TCP Connection Issue

Ends up the rabbitmq logs held the answer. I saw that something was wrong, but the messages were so cryptic I couldn't tell what was actually breaking. Turns out it was an SSL issue caused by an old ...
user avatar
  • 313
5 votes

monitoring/alerting system which can support very arbitrary data source and logic

Adding my brick to the list here, I'm at a point where updating continuously the monitoring system to add new instances and cleanup old ones is a pain. As such I've turn toward prometeus which works ...
user avatar
  • 11k
5 votes
Accepted

Monitoring immutable infrastructure virtual machines

I've looked a bit at Consul and Prometheus and am thinking that that is the way forward? In short, yes. Consul is designed exactly for this - it will run health checks against all of your services ...
user avatar
5 votes

Health monitoring of web pages without using GET check

I see two way to solve this problem: Check your logs often and ensure there was an access with code 200 in the last N seconds/minutes and no code 5xx meaning there's a server side error. (the often ...
user avatar
  • 11k
4 votes

What tools do you use to monitor applications and business-specific data? (apart from Jenkins)

We use Datadog, with metrics pushed from important operational processes using the statsd protocol ( http://docs.datadoghq.com/guides/metrics/ ), so we aren't too deeply tied to Datadog on the ...
user avatar
4 votes
Accepted

How do you monitor Node Micro services running inside of Docker Containers?

This is an area where Kubernetes has the correct model, there should be a load balancer between all systems which should have functional health checks. Once you start to add Nagios, Zabbix or other ...
user avatar
  • 366
4 votes

How do you monitor Node Micro services running inside of Docker Containers?

One commonly used solution(not container-specific) is to build a health check API within your service which tests all the functionality you want monitored (say availability of DBs and other ...
user avatar
  • 324
4 votes
Accepted

What opensource solutions can be used to implement anomaly detection based on Riemann metrics?

I can think of 3 possibility (Not exactly Riemann metrics based, but around anomaly detection): Outdated but the most efficient in anomaly detectionI know: Shyline + Oculus from Etsy Both tools ...
user avatar
  • 11k
4 votes
Accepted

Is Serf an added value when a container orchestration platform like kubernetes is used?

Serf is gossip protocol implementation and has absolutely nothing to do with orchestration tools like k8s. Even if such protocol is used internally, then in most cases it isn't available for you ...
user avatar
  • 446
4 votes

Monitor program progress on multiple servers

Graylog Since two people already advised you to rethink your current process (which I second since it will cause you sleepless nights at some point ;)), I will go another route and recommend a ...
user avatar
  • 140
4 votes

Infrastructure Stability Report

Its very common to provide a StatusPage (like Statusy.co or StatusPage.io). There are numerous examples of major providers having them: https://status.github.com/messages https://status.bitbucket.org/...
user avatar
  • 1,322
4 votes

What is a good strategy to reliably and automatically perform remote system tests of a web service?

For less trafficked parts of your web service, synthetic transactions are a good option. There are some trade-offs to be made when choosing either using an API to or using a full browser style check. ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible