7

This article about "How healthy is your Dockerized application?" explains the trouble with monitoring, but it doesn't provide any good examples of how to actually monitor a microservice inside of the docker container.

We are currently using PM2 monit to monitor our microservices, but as we put them into docker containers we lose the ability to access this data within one screen for all the various microservices which each run in their own docker container.

Dockerswarm monitoring will tell us the state of the containers, but not the microservice running inside of them.

What's a solid proven way of solving this problem?

4

This is an area where Kubernetes has the correct model, there should be a load balancer between all systems which should have functional health checks.

Once you start to add Nagios, Zabbix or other types of monitoring to the system you start to build a large state machine. This will break the loose coupling model and introduce inter-dependencies that inhibit the ease refactoring. While not set in stone the key differentiation between microservices and other variants of SOA is this loose coupling.

If the services are fine-grained and perform a single function, implement a health check at an upstream load balancer, then monitor the active pool members.

As an example in HAproxy

backend myapp
[...]
option tcp-check
tcp-check send GET\ /health HTTP/1.0\r\n
tcp-check send Host:\ foo\r\n
tcp-check send \r\n
tcp-check expect rstring ^HTTP/1.1\ 200\ Ok
tcp-check expect string container\ Good
server srv1 10.0.0.1:8080 check
server srv2 10.0.0.2:8080 check

In theory you don't care about the performance of an actual container, just that your overall performance is good.

This method makes it easy to have the system self repair and to scale with a minimal amount of complexity.

Basically you only have to check if the number of systems you expect are alive, and if not you spin up some more. If you need to add capacity you simply change the number of expected nodes.

This also simplifies refactoring as you only need to replicate or modify this test with no external dependencies or state machine.

It should also reduce down time and middle of the night Pagerduty alerts as the system self repairs.

As for the overall systems metrics, which are needed to trace down issues like latency I would want them in a central location using a tool like elasticsearch. If you use syslog, logstash or log4??? to collect metrics that will be far more useful in the long run. When systems are small and simple traditional polling based monitoring may provide enough metrics but it is preferable to have them in a format that is searchable and more importantly relational to other systems.

Solutions like monit still have their place, but it is to monitor the long lived components like the VMs or bare metal hosting your swarm, but the containers themselves should be decoupled from that system to get the most benefits from a micro-services model.

  • Why don't I care about the performance on a specific container? Isn't that the easiest way to figure out where the bottleneck is, and also to know when to scale out? – avi May 21 '17 at 7:38
  • You can track the performance of the container host, which is the same thing. But really it is a question if you want to follow the microservices model or not. While it is not the only option to get work done, loose coupling is one of the core tenants of the model – gdahlm May 21 '17 at 20:11
  • @gdahim Sorry, my question wasn't clear. I'm asking how that causes coupling. I see the benefit of the health check, but your answer doesn't make it clear to me why I don't care about cpu or mem usage in the container. – avi May 22 '17 at 7:15
  • 1
    There are multiple reasons but here are few: 1) Agent versions and configurations will need to be synchronized across the environment which will require coordinated efforts. 2) Any changes or refactoring of the internal system will potentially produce alerts or reductions in monitoring which may result in a need to coordinate changes. 3) It increases the required services per container which will require integration testing or gates. But also remember that a container is a namespace, and not a separate entity, monitoring the container host will produce the same numbers. – gdahlm May 22 '17 at 17:21
4

One commonly used solution(not container-specific) is to build a health check API within your service which tests all the functionality you want monitored (say availability of DBs and other dependencies) and the app itself, and returns some expected output (say < service >: < status >). You can then trigger alerts from a monitoring service like Nagios if this API does not return an ok for all the services. This will also fail if the microservice itself is unhealthy.

This approach also has the benefit of running a functional test of your service (by hitting an API end point).

This approach does not cover some edge cases though - eg. the microservice runs (but particular APIs fail).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.