Health monitoring of web pages without using GET check

Question

I am looking for a solution to monitor the health (DOWN or UP) of my web pages. Let's say I have three services,

site.com/data
site.com/user
site.com/id

Now, what I can do is that I can create a GET request which will be served every second and check if response status code is 200 OK. If it is anything else than 200 OK, then it will trigger an alert. But, I don't want to hit the server continuously, as it will create unnecessary logs. I am using a mongodb logger. What is the recommended way to achieve this?

Answer 1

I see two way to solve this problem:

1. Check your logs often and ensure there was an access with code 200 in the last N seconds/minutes and no code 5xx meaning there's a server side error. (the often should be coherent with how long you are ok with no entries)

2. Keep an active check but:

   • use a HEAD request (so there's less data to return by the server)
   • use a custom user agent for the check
   • configure your log exporter to ignore this user agent.
   • take a larger check interval, 1 second is probably over obsessive

Depending on how often your sites are called you may wish to mix both methods to have a proper check even if there's no traffic from real clients.

Answer 2

One possibility would be to monitor the responses to requests from other site users. For example I'd check if there were 200 OK responses and skip hitting the server is such responses were sent in the most recent X minutes (up to you to determine which time interval is best fit for your application).

Maybe checking on the (server side) errors as well and hitting the server as well if a sudden change in the error occurrence rate is detected - just to be sure.

Answer 3

While I like the other great answers already here, I'd like to add:

use GET against a different port

You could have your app listen on a different port which isn't logged. haproxy and such are happy to check for health on port A for a service that is live on port B.

don't log canary checks

You could designate certain data items for monitoring and other tests to retrieve and not log requests against those designated items.

set a lower TTL for canary entries

It is a bit scary to cut off logging, so maybe you could delete the entries related to logging more aggressively since it is going into a database? This avoids checking a code path that turns out to not fail when the user experience is failing.

end-to-end checks are pretty important

Having something that is able to

• see whether it can get to you
• and whether your thing is doing its thing
• and whether your thing can talk to everything it needs

often ends up being pretty vital to detect issues that you wouldn't catch by focusing on any one part of this. Paying the price for logging is usually negligible compared to the value and peace of mind you get from end-to-end checks.

Answer 4

You can use Uptime Robot, it offers a very good free layer (up to 50 sites free).

You can configure your sites (they are called monitors) in many ways. A simple GET (your case), ping and more. In case you want to have an idea, the screen to add a monitor look like this:

You can also configure your email to receive alerts, it has more options but I haven't explored which one of them are on the free tier, but in the other options you can find SMS, slack, hipchat and more.

It has a very good dashboard (I never use it though, I just get the email notifications) where you can see the status of the health checks and the type of error if it's the case.

The only downside on the free tier is the five minutes interval, I would prefer to use a two or three minutes interval but most of the sites don't need one second accuracy on the health check anyway.

Answer 5

One method is that you do TCP-only monitoring. In this scenario, you establish a three way handshake (or use tcp-half-open monitoring in which no connection is established - you merely look for an [syn, ack] to your [syn], and don't [ack] the [syn, ack] back) and then close the connection assuming it is the web server listening on that port.

A better option however, is to figure out how to exclude your health monitor traffic from logging, though I am unable to give specifics on doing that with rails/mongodb logger.