How do you monitor status of multiple docker containers?

Question

My home server has about 15 distinct services running, each represented by a Docker container. Depending on how the developers implemented the Dockerfile, some services have health checks, but most do not.

At the moment, I do not have a way to make sure these services remain healthy and running. I do have the restart: always setting set, but worst case this just means Docker endlessly restarts a server if it fails to start or crashes on startup for whatever reason. Or maybe a docker pull caused an issue during an update. There could be many reasons why these services do not start, or stop running during the day.

What I'd like is to know when they go down. Maybe an email that notifies me, so I can go fix the issue, instead of only finding out about the issue when a family member notifies me or when I try to use that service.

Is there a reasonable way to monitor multiple Docker containers in this way? Do I have to hand-write a bunch of scripts that do ps to check for process up time or something? Preferably an out-of-the-box solution would be great. Note I use Docker Compose for everything, and I only have 1 physical machine where I run these services, so I don't need to worry about complex setups.

Answer 1

Note: answer based only on docs, I didn't play with docker much.

If you choose scripting you could pick a higher layer than just ps, based on either:

• the docker-compose events CLI:

Usage: events [options] [SERVICE...]

Options:
    --json      Output events as a stream of json objects

Stream container events for every container in the project.

• the docker events CLI:

Description

Get real time events from the server

You'd be looking for the die event of the container(from Extended description):

Use docker events to get real-time events from the server. These events differ per Docker object type.

Object types

CONTAINERS

• die

The event appears to be generated regardless of the restart policy (from Docker Events Explained):

But you're right, there's a ton of monitoring tools out there, I also suspect some already do what you're seeking.

Just aimlessly wandering through those links I got to Implementing Docker event monitoring from scratch:

What else? I can create alerts on them. The next time any of my containers decide to puke and die, I can get an SMS message about their doomed state.

Answer 2

If you don't mind running a few other services at your home, then I'd suggest you try Prometheus with Grafana, dockerd-exporter and AlertManager.

Prometheus will store time series data collected from your dockerd-exporter, Grafana will display that data in a very nice way and AlertManager will allow you to configure alerts based on your needs.

Check out a random screenshot I found on Google that presents the solution:

Also check out swarmprom project.

Answer 3

If most of those are network services, or at least have some administrative TCP/IP port open, then monitoring them by curl or netcat might not be the worst solution. That's how a full-fledged orchestration solution like OpenShift would do it. So, in somewhat increasing order:

• Check if a TCP/IP port is open (netcat -z)
• Check if the port is open and responds with a specific prompt (netcat | grep HELLO).
• Check if the port answers with a specific HTML (for example) (curl ... | grep <p>I'm alive</p>).

And finally, if nothing else helps:

• Check if a hand-written script returns true.

Answer 4

You can use check_mk for monitoring the health of containers just as you could for any other machine. Read about it here..

Open Source Docker Monitoring Using Check MK

Hope this helps.