Monitoring checklist - What things should I be monitoring?

Question

We are building a (Zabbix-based) monitoring system for our applications; hovewer, I'm having difficulties in defining what to monitor?

I have so far come up with the following general categories:

• hardware data: cpu, ram, swap, etc.
• middleware data: perfomance/health for MySQL instantces, Tomcat instances, JVMs, etc.
• logical or application data: the current status/health of the system, e.g. number of active users, page request, etc.
• kpi data: data for business, e.g. user registration over time.
• dashboard: quick overview of the system (e.g. microservices are running or not).

Are there any other fundamental categories for to monitor? Or is there another category system to use?

UPDATE: the purpose of the monitoring is

• the see if the system functions correctly (at high-level, e.g. no services are down, etc. - much like a smoke-test)
• see, if there are any indicators, that the system is likely to crash (e.g. historical data predicts that we will run out of disk space)
• if any of these occur, send a warning to the appropriate staff (e.g. via e-mail)

UPDATE: the complexity of our system does not demand an extra application for reporting (e.g. monitoring KPIs); also, we are running in local/local cloud infrastructure, so the cost of the application is not (that)relevant - but it might be someday :-)

Answer 1

I like this video: GOTO 2016 • Monitoring Microservices • Tom Wilkie

One of the key ideas (for me at least) is to realize the difference between host monitoring and application monitoring. Basically host monitoring tells you that something is fatally wrong now, but application monitoring should be able to predict problems by detecting higher error rate or that requests are taking longer time so you can fix problems before your users notice them.

(I'm not affiliated with weaveworks or the goto conference in any way, I just like the content and think there are some interesting ideas. Use the downvote button to let me know that this answer is not good :) )

Answer 2

I'm surprised nobody mentioned the four golden signals explicitly as an answer, so I'll add it. Lifted directly from Google's SRE Book chapter on monitoring distributed systems, it is suggested to at least collect metrics on the "Four Golden Signals":

1. Latency:
   The time it takes to service a request. It’s important to distinguish between the latency of successful requests and the latency of failed requests. For example, an HTTP 500 error triggered due to loss of connection to a database or other critical backend might be served very quickly; however, as an HTTP 500 error indicates a failed request, factoring 500s into your overall latency might result in misleading calculations. On the other hand, a slow error is even worse than a fast error! Therefore, it’s important to track error latency, as opposed to just filtering out errors.
2. Traffic:
   A measure of how much demand is being placed on your system, measured in a high-level system-specific metric. For a web service, this measurement is usually HTTP requests per second, perhaps broken out by the nature of the requests (e.g., static versus dynamic content). For an audio streaming system, this measurement might focus on network I/O rate or concurrent sessions. For a key-value storage system, this measurement might be transactions and retrievals per second.
3. Errors:
   The rate of requests that fail, either explicitly (e.g., HTTP 500s), implicitly (for example, an HTTP 200 success response, but coupled with the wrong content), or by policy (for example, "If you committed to one-second response times, any request over one second is an error"). Where protocol response codes are insufficient to express all failure conditions, secondary (internal) protocols may be necessary to track partial failure modes. Monitoring these cases can be drastically different: catching HTTP 500s at your load balancer can do a decent job of catching all completely failed requests, while only end-to-end system tests can detect that you’re serving the wrong content.
4. Saturation:
   How "full" your service is. A measure of your system fraction, emphasizing the resources that are most constrained (e.g., in a memory-constrained system, show memory; in an I/O-constrained system, show I/O). Note that many systems degrade in performance before they achieve 100% utilization, so having a utilization target is essential. In complex systems, saturation can be supplemented with higher-level load measurement: can your service properly handle double the traffic, handle only 10% more traffic, or handle even less traffic than it currently receives? For very simple services that have no parameters that alter the complexity of the request (e.g., "Give me a nonce" or "I need a globally unique monotonic integer") that rarely change configuration, a static value from a load test might be adequate. As discussed in the previous paragraph, however, most services need to use indirect signals like CPU utilization or network bandwidth that have a known upper bound. Latency increases are often a leading indicator of saturation. Measuring your 99th percentile response time over some small window (e.g., one minute) can give a very early signal of saturation. Finally, saturation is also concerned with predictions of impending saturation, such as "It looks like your database will fill its hard drive in 4 hours."

If you measure all four golden signals and page a human when one signal is problematic (or, in the case of saturation, nearly problematic), your service will be at least decently covered by monitoring.

Answer 3

It depends a lot on what your infrastructure situation is. If you're doing auto-scaling, the health of individual instances is mostly irrelevant. The important metrics are total cost, and cost per unit of work (e.g. per request). Personally I don't like to monitor individual instance state if I can possibly avoid it - I try to focus more on broader service-level and application-level metrics:

• Overall per-service uptime (% of the time that at least one instance is healthy and able to respond to new requests immediately)
• Overall end-user uptime (% of the time that the user perceives the product as being available - if some core service goes down, this might be user-facing downtime, but a lesser-used service or background worker might not be)
• 95th percentile response time of each service
• Message queue length and net queue growth per queue
• Consumer count per queue
• Average time to completion per queue (not just time in queue, but time from being queued to completed work for the message)
• Error rate for each service
• Cost per unit of work for each service
• For each server role, what is the ratio of the average CPU utilization to average memory utilization? This is useful for determining if we're scaling poorly - if CPU use is 70% but memory is 20%, we're giving instances too much memory, and vice-versa.
• Same for peak CPU/memory utilization
• Time to delivery for deployments

Some of your listed metrics like user registration over time, to me, don't belong in an infrastructure monitoring system like Zabbix. What is anyone watching Zabbing going to do about a 10% drop in registrations? Nothing. This is business reporting data that should be exposed to whoever wants it via a reporting DB, possibly rendered in a nice dashboard because pointy-haired bosses love dashboards.

Answer 4

For any given node, there are 4 basic resources with the following items to monitor:

• CPU
  • Total
  • Per-core
• Storage
  • Free space
  • Free inodes
  • Throughput
  • Backups
• Memory
  • Free
  • Swap
  • Buffers
  • Cache
• Network
  • Latency
  • Throughput
  • Jitter

These four basic resources will power your application or service at every layer of your application. This will be architected into up to 5 layers depending on your environment and you will want to monitor them at every layer. This might look like:

This can change depending on how your environment is architected. For example, if you run all of your data off of NFS mounts, the Storage layer would sit beside compute instead of behind it. If you have a C++ based application that workstations connect to, you might not have a front-end layer. If your application uses flat files, you might not have a database layer. If you do not use virtual machines or containers and use local storage, you might not have a compute and storage layer. You will want to monitor the above 4 basic resources at every layer.

This represents the basic monitoring that can be given by your vendors, software and hardware. Covering these four basic resources at every layer should be your first goal. Once 100% coverage is achieved, You can additionally build hooks into your application to report additional health statuses, but by their nature, these will typically be built as a reaction to outages and you would have to work with your internal developers to build these kinds of hooks.

Monitoring these 4 basic resources should catch probably 80% of issues that cause outages however, and then you can start working on the remaining 20%.