How do you monitor Node Micro services running inside of Docker Containers?

Question

This article about "How healthy is your Dockerized application?" explains the trouble with monitoring, but it doesn't provide any good examples of how to actually monitor a microservice inside of the docker container.

We are currently using PM2 monit to monitor our microservices, but as we put them into docker containers we lose the ability to access this data within one screen for all the various microservices which each run in their own docker container.

Dockerswarm monitoring will tell us the state of the containers, but not the microservice running inside of them.

What's a solid proven way of solving this problem?

Answer 1

This is an area where Kubernetes has the correct model, there should be a load balancer between all systems which should have functional health checks.

Once you start to add Nagios, Zabbix or other types of monitoring to the system you start to build a large state machine. This will break the loose coupling model and introduce inter-dependencies that inhibit the ease refactoring. While not set in stone the key differentiation between microservices and other variants of SOA is this loose coupling.

If the services are fine-grained and perform a single function, implement a health check at an upstream load balancer, then monitor the active pool members.

As an example in HAproxy

backend myapp
[...]
option tcp-check
tcp-check send GET\ /health HTTP/1.0\r\n
tcp-check send Host:\ foo\r\n
tcp-check send \r\n
tcp-check expect rstring ^HTTP/1.1\ 200\ Ok
tcp-check expect string container\ Good
server srv1 10.0.0.1:8080 check
server srv2 10.0.0.2:8080 check

In theory you don't care about the performance of an actual container, just that your overall performance is good.

This method makes it easy to have the system self repair and to scale with a minimal amount of complexity.

Basically you only have to check if the number of systems you expect are alive, and if not you spin up some more. If you need to add capacity you simply change the number of expected nodes.

This also simplifies refactoring as you only need to replicate or modify this test with no external dependencies or state machine.

It should also reduce down time and middle of the night Pagerduty alerts as the system self repairs.

As for the overall systems metrics, which are needed to trace down issues like latency I would want them in a central location using a tool like elasticsearch. If you use syslog, logstash or log4??? to collect metrics that will be far more useful in the long run. When systems are small and simple traditional polling based monitoring may provide enough metrics but it is preferable to have them in a format that is searchable and more importantly relational to other systems.

Solutions like monit still have their place, but it is to monitor the long lived components like the VMs or bare metal hosting your swarm, but the containers themselves should be decoupled from that system to get the most benefits from a micro-services model.

Answer 2

One commonly used solution(not container-specific) is to build a health check API within your service which tests all the functionality you want monitored (say availability of DBs and other dependencies) and the app itself, and returns some expected output (say < service >: < status >). You can then trigger alerts from a monitoring service like Nagios if this API does not return an ok for all the services. This will also fail if the microservice itself is unhealthy.

This approach also has the benefit of running a functional test of your service (by hitting an API end point).

This approach does not cover some edge cases though - eg. the microservice runs (but particular APIs fail).