How to monitor ELK Stack Load?

Question

We have tried Prometheus and Grafana. Currently it is not set up on our cluster and we are tight with resources.

• What are the best ways to monitor load on an ELK stack?
• Is there anything built in?

Answer 1

Load can mean a lot with an ELK stack.

For ingestion load, the logstash parsing side, you're looking at a combination of one or two things:

• CPU load on the ingestion boxes.
• If used, back-pressure in your ingestion-queue (such as Redis, AMQP, or others)
  • This can be simple stuff like queue-size, or a fancy number like age of the oldest member of the queue.

If you're in a cloud-provider that gives you a way to auto-scale your ingestion nodes, keying on CPU functions pretty well in recent Logstash versions.

For ingestion and searching, the ElasticSearch side, it can depend on a couple of things due to your architecture. If you're using physical servers, it is very easy to have far more CPU capacity than I/O capacity to store the processed results. If you're using physical servers, keep an eye on I/O queue depth in addition to CPU loading. Also beware of the Compressed Object Pointer barrier, around 32GB of JVM HEAP. Stay under 32GB. Just do it.

If you're on cloudy instances, its harder to out-strip your I/O due to the reduced core-counts. Even so, queue-depth and cpu-loading are good things to keep an eye on. If your JVM monitoring tools are up to it, tracking Garbage Collection frequency is worth it until you have a feel for what you need for RAM.

For all of this, keeping an eye on consumed disk-space will tell you when you need to add more ES data-nodes or adjust your index retention settings.

For display and searching, such as with Kibana or Grafana, I've found this to be the part of the stack I never bother looking at. At least in any way other than up/down tracking. We track CPU, but we don't alarm anything since in our experience even a 2 core box with 7GB of RAM is enough to keep up without breaking a sweat. Both of these work OK with horizontal scaling if you do run into problems.

---

ELK stacks come in many sizes, from all in one boxes, all the way up to systems with routing logic to bring events to parsing and ES clusters owned by completely different parts of the organization. The monitoring needs of a small architecture (all in one) are rather different than medium architectures (discrete parsing tier, usually queue-mediated), and are different again from large architectures (multiple parsing-tiers with multiple event-stores).

Answer 2

I'm using Datadog which has a really great EKS plugin that works really well with the other AWS integrations.

Downside is cost, which is $15 per node on the basic plan.