28

We have an application that writes three types of logs into three separate files: access logs, generic application logs and system logs. The format (and purpose) of these logs are very different. And we have separate logforwarders that send them separately to our centralised logging system.

Based on the treat logs as event streams principle, we are thinking about moving from using files to stdout. While we know some of the benefit of this approach, this would also mean that we'd get a merged stream of the differently formatted logs, which we would need to split again either before we can send them to our central system (Kibana / Splunk / etc.), or inside there.

We're wondering whether there are any tools or recommendations on how we should approach this situation.

Improve this question
8
  • 4
    I don't think it's worth it. Why work harder to merge and then split log streams, just because of some "principle"? Files are good. Files work. This sounds overengineered. I would say maybe pipe all logs into syslog, with different tags, etc. but I must say, if someone in my team suggested this.. I would be .. disappointed.
    – Assaf Lavie
    Commented Mar 7, 2017 at 16:22
  • Because using files have some other types of management nightmares, especially if they are generated from inside a docker container. For now it looks like the disadvantages from switching to stdout outweigh the benefits for our use case, but we are already having trouble with our file based approach
    – SztupY
    Commented Mar 7, 2017 at 16:24
  • 3
    I don't know about "nightmares". I know that this is they way it's been done for a while now, and there's lots of software out there that helps you do this. Log files are rotated, read with checkpoints - a file is a great abstraction for this. I wouldn't buy into a principle that sells me new nightmares because of fear of old, familiar patterns. Your log records are either written to a file, or handled in memory (at least as long as they're being moved around in the container). Good luck achieving the reliability of log files with in-memory stream splitters & mergers.
    – Assaf Lavie
    Commented Mar 7, 2017 at 16:28
  • @AssafLavie You should write it in an answer that can get upvoted. IMHO it's a perfectly valid point of view.
    – Dan Cornilescu
    Commented Mar 7, 2017 at 16:35
  • 2
    I came here with the same question. The simple fact is that docker's built-in logging functionality depends on everything going to stdout/stderr, hence the logging driver and the ecosystem of 3rd party tools building up around it does too. I'm tempted to dump all my logs in a host volume too, but I know I'm going to have to keep going back and managing that when my containers move to k8s or openshift or gke or whatever, whereas if I follow the docker stdout approach it'll be much smoother. Meanwhile I'll keep looking for an answer to this legitimate question
    – Rhubarb
    Commented Jul 28, 2017 at 9:56

2 Answers 2

Reset to default
16

I'm still looking for a merging/splitting approach myself, but meanwhile this approach recommended by Kubernetes documentation seems like a sound solution: Use a sidecar container for each of your separate logs.

A "sidecar" is any docker container that you use alongside another docker container to work with it in some way. In this case for each of your three logs you would have a separate container that scans or tails the logs and outputs to stdout.

This way each of your log-sidecar-containers has its own docker-log from its own stdout. Being separate like this, you can use standard docker (and kubernetes, etc) practices for separating or aggregating. Here's what the Kubernetes page has to say:

This approach allows you to separate several log streams from different parts of your application, some of which can lack support for writing to stdout or stderr. The logic behind redirecting logs is minimal, so it’s hardly a significant overhead. Additionally, because stdout and stderr are handled by the kubelet, you can use built-in tools like kubectl logs.

The "separate log streams" stem from the built-in tagging that docker applies to logs from different containers, described in the docker documentation here:

The tag log option specifies how to format a tag that identifies the container’s log messages. By default, the system uses the first 12 characters of the container id. To override this behavior, specify a tag option

Improve this answer
1
  • It is worth mentioning the downside of this log-sidecar-containers approach, quote : "Note, that despite low CPU and memory usage, writing logs to a file and then streaming them to stdout can double disk usage". I wonder if anyone tries this approach in practice.
    – yusong
    Commented Oct 22, 2019 at 17:16
8

The idea of merging them into one stream just to split them later sounds painful. I've not had a reason to do this myself, but here's where I'd start:

  • When you run the docker container, create a volume such that it will be easy to view/ship logs from the host.
  • Use something like remote_syslog2 to ship the logs to your log collector

It feels a little less-than-elegant to have to do some setup on the host as well, but if you use something like ansible where you can run a playbook and set that up during your deploy to the box it shouldn't be too bad.

Improve this answer
1
  • This solution can be combined with named pipes, the only problem with named pipes is that they don't work on all systems right now. They don't work on Mac
    – Jens
    Commented May 27, 2019 at 2:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.