14

I think what you are looking for is an Open Source project that can take inputs from both Amazon CloudWatch and Google StackDriver; there isn't a huge amount out there at the moment, but I will detail what I know. I have made the assumption that you know how to import your application telemetry into the solutions below. Open Source Open source solutions ...


11

The fire hose is a reference to an old Weird Al Yankovick movie, UHF (1989). In this movie, George Newman (portrayed by Yankovick) starts a local TV channel in the UHF band and is responsible for programming on the channel. After befriending the Station's Janitor, Stanley Spadowski, (portrayed by Michael Richards) George gives the Janitor a slot to run a ...


9

All in all any monitoring tool would do for this case, monit, nagios, shinken, icinga, centreon, or even a crontab in bash could do...


8

Have a look at the Monitoring plugin. Here are some more details about it (from the linked page): Charts of memory, cpu, system load average, http response times by day, week, month, year or custom period Statistics of http requests with mean response times, mean cpu times, mean response size by request and by day, week, month, year or custom period ...


8

I'm not sure it's a best practice but what I would do is using filebeat to read the log in near real time and push it to ElasticSearch. AWS provides an ElasticSearch Service which can save you from this part. The drawback of using filebeat directly to elasticsearch is that the time of events will be the time the prospector read them and if your logs have a ...


8

Any time you're running ad-hoc long-running commands, you should step back and rethink your process, because that should be automated, including error handling. Rather than connecting in to the servers to see status, a better approach is to push that information out. You can do a wide variety of things if your want to write a bunch of custom code, but the ...


7

I have no experience with it myself, but Elastalert (http://elastalert.readthedocs.io/en/latest/elastalert.html) sounds like what you need (for the alarms/threshold part) But besides that, it sounds like your interested in monitoring. In my opinion ELK is not a classic monitoring tool (more: data collection, data processing and data visualization). You ...


6

Consider using something like nagios or Zabbix. While it might be nice to avoid custom queries and alerts, this is impossible. The odds of someone using your specific suite of tools in your configuration are slim-to-none - and the odds of having pre-canned monitoring for that suite in that specific configuration are even slimmer. Simply put, there is no way ...


6

As far as I know, Prometheus doesn't mind high-cardinality data. What Prometheus doesn't like is high-cardinality labels. Let's start with Prometheus official documentation, it gives a good high-level explanation why: CAUTION: Remember that every unique combination of key-value label pairs represents a new time series, which can dramatically increase ...


5

I think your can do some kind of alerting on a metric rate with something like this: ALERT DropInMetricsFromExporter IF rate(<metric_name>[1m]) == 0 FOR 3m ANNOTATIONS { summary = "Rate of metrics is 0 {{ $labels.<your_label> }}", description = "Rate of metric dropped, actually: {{ $value }}%", } The main idea is to alert whenever ...


5

I've started abusing SyncThing for this purpose (among others), configuring it as a system service on laptops, then locking down it's UI to prevent it from being able to be used by a local connection originator to manipulate or access files. I get connectivity monitoring and robust rsync-like backup of field data. It's fantastic at punching through ...


5

Adding my brick to the list here, I'm at a point where updating continuously the monitoring system to add new instances and cleanup old ones is a pain. As such I've turn toward prometeus which works with a simple contract, an exporter does the collection job (could be the cloudwatch_exporter on AWS or when using netdata querying it (either a central one or ...


5

I've looked a bit at Consul and Prometheus and am thinking that that is the way forward? In short, yes. Consul is designed exactly for this - it will run health checks against all of your services that you define and can detect when things go down. However, what it does not do it alert when things do go down, it is more designed for automatically taking ...


5

I see two way to solve this problem: Check your logs often and ensure there was an access with code 200 in the last N seconds/minutes and no code 5xx meaning there's a server side error. (the often should be coherent with how long you are ok with no entries) Keep an active check but: use a HEAD request (so there's less data to return by the server) use a ...


5

Prometheus use a timeseries databases with vacuum, the documentation gives some maths to plan your disk consumption: On average, Prometheus uses only around 1-2 bytes per sample. Thus, to plan the capacity of a Prometheus server, you can use the rough formula: needed_disk_space = retention_time_seconds * ingested_samples_per_second * bytes_per_sample ...


4

Ends up the rabbitmq logs held the answer. I saw that something was wrong, but the messages were so cryptic I couldn't tell what was actually breaking. Turns out it was an SSL issue caused by an old version of Erlang, which is what apt was installing by default on ubuntu 14.04 (erlang version R16B03). This amqp issue was what pointed me to the solution: ...


4

We use Datadog, with metrics pushed from important operational processes using the statsd protocol ( http://docs.datadoghq.com/guides/metrics/ ), so we aren't too deeply tied to Datadog on the application side. In Datadog's interface we can then configure monitors to alert on any number of things, including anomaly detection if defining thresholds is ...


4

This is an area where Kubernetes has the correct model, there should be a load balancer between all systems which should have functional health checks. Once you start to add Nagios, Zabbix or other types of monitoring to the system you start to build a large state machine. This will break the loose coupling model and introduce inter-dependencies that ...


4

One commonly used solution(not container-specific) is to build a health check API within your service which tests all the functionality you want monitored (say availability of DBs and other dependencies) and the app itself, and returns some expected output (say < service >: < status >). You can then trigger alerts from a monitoring service like Nagios ...


4

As for workaround, the actual memory can be checked by invoking Groovy commands directly in Script Console (at /computer/(master)/script). Example command: println "free -m".execute().text


4

I can think of 3 possibility (Not exactly Riemann metrics based, but around anomaly detection): Outdated but the most efficient in anomaly detectionI know: Shyline + Oculus from Etsy Both tools available from etsy's github as Skyline and oculus, check the network graph for more up to date forks. Best one I found is earthgecko/skyline with its doc Graphite ...


4

Serf is gossip protocol implementation and has absolutely nothing to do with orchestration tools like k8s. Even if such protocol is used internally, then in most cases it isn't available for you applications running in given orchestrator. So if your app or workflow depends on gossip protocol, then you will have use out of Serf, otherwise there is no point. ...


4

Graylog Since two people already advised you to rethink your current process (which I second since it will cause you sleepless nights at some point ;)), I will go another route and recommend a specific piece of software which - in my opinion - fits most of your needs: Graylog. I implemented and used a couple of ELK stacks for both log aggregation and ...


4

Its very common to provide a StatusPage (like Statusy.co or StatusPage.io). There are numerous examples of major providers having them: https://status.github.com/messages https://status.bitbucket.org/ https://status.aws.amazon.com/ https://azure.microsoft.com/en-us/status/ https://api.twitterstat.us/ You could provide the same status page to your customers....


3

I like this video: GOTO 2016 • Monitoring Microservices • Tom Wilkie One of the key ideas (for me at least) is to realize the difference between host monitoring and application monitoring. Basically host monitoring tells you that something is fatally wrong now, but application monitoring should be able to predict problems by detecting higher error rate or ...


3

A third party app metrics tools is great if it already covers all (or most of) the parameters you want to monitor and it presents the information in a convenient enough manner for your use case - you just need to (find it first and then) learn how to install/use/maintain it and then do that. Advantages: avoids the software development and maintenance costs,...


3

If your Zabbix agent is 2.2 or later, you could use Zabbix agent item vfs.file.regexp like so: vfs.file.regexp[/proc/meminfo,^Dirty.*([0-9]+),,,,\1] If your Zabbix is 3.4 or later, you can use a master item to collect the data blob (perhaps with vfs.file.contents) and then extract and parse the desired values with item value preprocessing.


3

As Tensibai mentioned, you can extract this info from the /proc filesystem, but in most cases you need to determine the trending yourself. There are several places which could be of interest: /proc/[pid]/statm Provides information about memory usage, measured in pages. The columns are: size (1) total program size ...


3

I would suggest elastalert as an alert system for elasticsearch, it has amazing capabilities, you can execute any query on elasticsearch and define an alarm depending on the response and it support multi ways for alert like sending email or slack or telegram, even execute specific commands. For monitor data in elasticsearch as you are already using ELK, you ...


3

I agree with @Xiong Chiamiov and I want to give more clarify option. If you want every line in the CLI to be monitored, I would suggest to redirect all the output to specific file and the error to another file, then use logstash or filebeat to send both of this files to Elasticsearch, then you can configure Logtril with Kibana to give you view, analyze, ...


Only top voted, non community-wiki answers of a minimum length are eligible