14

I think what you are looking for is an Open Source project that can take inputs from both Amazon CloudWatch and Google StackDriver; there isn't a huge amount out there at the moment, but I will detail what I know. I have made the assumption that you know how to import your application telemetry into the solutions below. Open Source Open source solutions ...


12

As far as I know, Prometheus doesn't mind high-cardinality data. What Prometheus doesn't like is high-cardinality labels. Let's start with Prometheus official documentation, it gives a good high-level explanation why: CAUTION: Remember that every unique combination of key-value label pairs represents a new time series, which can dramatically increase ...


11

The fire hose is a reference to an old Weird Al Yankovick movie, UHF (1989). In this movie, George Newman (portrayed by Yankovick) starts a local TV channel in the UHF band and is responsible for programming on the channel. After befriending the Station's Janitor, Stanley Spadowski, (portrayed by Michael Richards) George gives the Janitor a slot to run a ...


10

Have a look at the Monitoring plugin. Here are some more details about it (from the linked page): Charts of memory, cpu, system load average, http response times by day, week, month, year or custom period Statistics of http requests with mean response times, mean cpu times, mean response size by request and by day, week, month, year or custom period ...


10

The CPUThrottlingHigh is an alert created by the kubernetes-mixin project. There is an open issue (#108) to discuss this alert. I suggest that you read all the comments on this issue to better understand the problem. In short, the problem is: When working with low CPU limits, spiky workloads can have low averages and still be being throttled. Also, take a ...


9

All in all any monitoring tool would do for this case, monit, nagios, shinken, icinga, centreon, or even a crontab in bash could do...


8

I'm not sure it's a best practice but what I would do is using filebeat to read the log in near real time and push it to ElasticSearch. AWS provides an ElasticSearch Service which can save you from this part. The drawback of using filebeat directly to elasticsearch is that the time of events will be the time the prospector read them and if your logs have a ...


8

Any time you're running ad-hoc long-running commands, you should step back and rethink your process, because that should be automated, including error handling. Rather than connecting in to the servers to see status, a better approach is to push that information out. You can do a wide variety of things if your want to write a bunch of custom code, but the ...


8

Prometheus use a timeseries databases with vacuum, the documentation gives some maths to plan your disk consumption: On average, Prometheus uses only around 1-2 bytes per sample. Thus, to plan the capacity of a Prometheus server, you can use the rough formula: needed_disk_space = retention_time_seconds * ingested_samples_per_second * bytes_per_sample ...


8

The term Alert Fatigue, also known as Alarm Fatigue has been around for decades. It affects many professions, including medical, technical, and construction industries. According to the Alert Fatigue by Other Names reference, the earliest literature on the topic came during the Israeli Arab conflict. The practice/understanding of the issue is ancient. ...


7

I have no experience with it myself, but Elastalert (http://elastalert.readthedocs.io/en/latest/elastalert.html) sounds like what you need (for the alarms/threshold part) But besides that, it sounds like your interested in monitoring. In my opinion ELK is not a classic monitoring tool (more: data collection, data processing and data visualization). You ...


6

Consider using something like nagios or Zabbix. While it might be nice to avoid custom queries and alerts, this is impossible. The odds of someone using your specific suite of tools in your configuration are slim-to-none - and the odds of having pre-canned monitoring for that suite in that specific configuration are even slimmer. Simply put, there is no way ...


6

I have spend years developing a variety of monitoring tools which employ a variety of approaches to track data points ranging from filtered logs, collecting system stats (i.e.: querying data from /proc or running a command and parsing output), and/or putting concise “tagging” code inside application workflow code to track overall performance. With all of ...


5

I like this video: GOTO 2016 • Monitoring Microservices • Tom Wilkie One of the key ideas (for me at least) is to realize the difference between host monitoring and application monitoring. Basically host monitoring tells you that something is fatally wrong now, but application monitoring should be able to predict problems by detecting higher error rate or ...


5

As for workaround, the actual memory can be checked by invoking Groovy commands directly in Script Console (at /computer/(master)/script). Example command: println "free -m".execute().text


5

I think your can do some kind of alerting on a metric rate with something like this: ALERT DropInMetricsFromExporter IF rate(<metric_name>[1m]) == 0 FOR 3m ANNOTATIONS { summary = "Rate of metrics is 0 {{ $labels.<your_label> }}", description = "Rate of metric dropped, actually: {{ $value }}%", } The main idea is to alert whenever ...


5

As Tensibai mentioned, you can extract this info from the /proc filesystem, but in most cases you need to determine the trending yourself. There are several places which could be of interest: /proc/[pid]/statm Provides information about memory usage, measured in pages. The columns are: size (1) total program size ...


5

I've started abusing SyncThing for this purpose (among others), configuring it as a system service on laptops, then locking down it's UI to prevent it from being able to be used by a local connection originator to manipulate or access files. I get connectivity monitoring and robust rsync-like backup of field data. It's fantastic at punching through ...


5

Ends up the rabbitmq logs held the answer. I saw that something was wrong, but the messages were so cryptic I couldn't tell what was actually breaking. Turns out it was an SSL issue caused by an old version of Erlang, which is what apt was installing by default on ubuntu 14.04 (erlang version R16B03). This amqp issue was what pointed me to the solution: ...


5

Adding my brick to the list here, I'm at a point where updating continuously the monitoring system to add new instances and cleanup old ones is a pain. As such I've turn toward prometeus which works with a simple contract, an exporter does the collection job (could be the cloudwatch_exporter on AWS or when using netdata querying it (either a central one or ...


5

I've looked a bit at Consul and Prometheus and am thinking that that is the way forward? In short, yes. Consul is designed exactly for this - it will run health checks against all of your services that you define and can detect when things go down. However, what it does not do it alert when things do go down, it is more designed for automatically taking ...


5

I see two way to solve this problem: Check your logs often and ensure there was an access with code 200 in the last N seconds/minutes and no code 5xx meaning there's a server side error. (the often should be coherent with how long you are ok with no entries) Keep an active check but: use a HEAD request (so there's less data to return by the server) use a ...


4

We use Datadog, with metrics pushed from important operational processes using the statsd protocol ( http://docs.datadoghq.com/guides/metrics/ ), so we aren't too deeply tied to Datadog on the application side. In Datadog's interface we can then configure monitors to alert on any number of things, including anomaly detection if defining thresholds is ...


4

This is an area where Kubernetes has the correct model, there should be a load balancer between all systems which should have functional health checks. Once you start to add Nagios, Zabbix or other types of monitoring to the system you start to build a large state machine. This will break the loose coupling model and introduce inter-dependencies that ...


4

One commonly used solution(not container-specific) is to build a health check API within your service which tests all the functionality you want monitored (say availability of DBs and other dependencies) and the app itself, and returns some expected output (say < service >: < status >). You can then trigger alerts from a monitoring service like Nagios ...


4

I can think of 3 possibility (Not exactly Riemann metrics based, but around anomaly detection): Outdated but the most efficient in anomaly detectionI know: Shyline + Oculus from Etsy Both tools available from etsy's github as Skyline and oculus, check the network graph for more up to date forks. Best one I found is earthgecko/skyline with its doc Graphite ...


4

Serf is gossip protocol implementation and has absolutely nothing to do with orchestration tools like k8s. Even if such protocol is used internally, then in most cases it isn't available for you applications running in given orchestrator. So if your app or workflow depends on gossip protocol, then you will have use out of Serf, otherwise there is no point. ...


4

Graylog Since two people already advised you to rethink your current process (which I second since it will cause you sleepless nights at some point ;)), I will go another route and recommend a specific piece of software which - in my opinion - fits most of your needs: Graylog. I implemented and used a couple of ELK stacks for both log aggregation and ...


4

Its very common to provide a StatusPage (like Statusy.co or StatusPage.io). There are numerous examples of major providers having them: https://status.github.com/messages https://status.bitbucket.org/ https://status.aws.amazon.com/ https://azure.microsoft.com/en-us/status/ https://api.twitterstat.us/ You could provide the same status page to your customers....


3

You need to collect static assets in your Django cd /opt/graphite/webapp && PYTHONPATH=/opt/graphite/webapp django-admin.py collectstatic --noinput --settings=graphite.settings Static files will be installed in /opt/graphite/static. Then you need to configure your webserver to serve them directly. For Nginx something like: location /static { ...


Only top voted, non community-wiki answers of a minimum length are eligible