What is a good logging practice for distributed tasks?

Question

I have the following setting:

Create multiple workers, do a computation and terminate them after the computation is done.

So, every-time it’ll be a different instance running the task, so each host will have its own a log file, this will result in a huge list of files.

Is it a good practice? If not, what would be a better way for logging the task processing in this particular use-case?

PS: My infrastructure is serverless. So, for now, I am logging to (AWS)CloudWatch. But, please answer the question independently of AWS, and suiting a serverless setup as much as possible.

Answer 1

"Serverless" mostly just means you've got relatively simple microservices, generally just a little webapp or a single function that is automatically connected to a REST frontend. The same concepts apply as you would use for a more traditional web services: usually some mix of remote syslog and ElasticSearch writers.

Networked or remote syslog has been around for a long time and has a fairly robust set of tools around it. You would have to run the central syslog server(s) but the protocol is very simple and there are pure client libraries in every language that you can use for sending logs. One common problem with remote syslog is that it has traditionally been based around UDP. This means that under heavy load, some log messages may be lost. This could be a good thing, helping avoid a cascade overload, but it is something to be aware of. Some newer syslog daemons also support a TCP-based protocol, but client support is less unified so just do your research.

More recent but very popular is logging to ElasticSearch. This is mostly useful because of the Kibana dashboard and Logstash tooklit (often called ELK, ElasticSearch+Logstash+Kibana). Amazon even offers a hosted ElasticSearch option, making it somewhat easier to get started. ES uses a relatively simple REST API, so any language with an HTTP client (read: all of them) should be okay with logging to ES but make sure you are careful with blocking network operations in cases of partial system outages (i.e. make sure your app won't get stuck in a logging call that will never succeed and stop servicing user requests).

More complex logging topologies are bounded only by your imagination, though these days you'll see a lot of use of the Kafka database/queue/whatever-you-want-to-call-it as a nexus point in very complex log distribution systems.

On the "serverless" side, you'll generally want to integrate with these systems directly at the network level, so sending log data directly to syslog or ES from your service/function, rather than writing to local files (though maybe echo to those too for local debugging and development).

Answer 2

This answer is more about scalability considerations - if the number of workers can be high and/or multiple of them can produce logs at high rate at the same time.

Yes, using multiple logfiles simultaneously is a good practice.

Attempting to combine into a single logfile logs from multiple workers in real time will raise problems:

• using blocking mechanims to prevent message loss will slow down the workers
• log messages can appear out-of-order in the combined logfile
• a centralized logging facility which combines the logs can be overloaded due to limited write speed, messages would be lost

Sharding logfiles (using multiple logfiles active in the same time) is itself a technique used by some hosting providers offering high performance, scalable centralized logging services. For example, when exporting logs to files Google's StackDriver Logging produces multiple sharded logfiles. From Log entries in Google Cloud Storage:

When you export logs to a Cloud Storage bucket, Stackdriver Logging writes a set of files to the bucket. The files are organized in directory hierarchies by log type and date. The log type can be a simple name like syslog or a compound name like appengine.googleapis.com/request_log. If these logs were stored in a bucket named my-gcs-bucket, then the directories would be named as in the following example:

my-gcs-bucket/syslog/YYYY/MM/DD/
my-gcs-bucket/appengine.googleapis.com/request_log/YYYY/MM/DD/

A single bucket can contain logs from multiple log types.

The leaf directories (DD/) contain multiple files, each of which holds the exported log entries for a time period specified in the file name. The files are sharded and their names end in a shard number, Sn or An (n=0, 1, 2, ...). For example, here are two files that might be stored within the directory my-gcs-bucket/syslog/2015/01/13/:

08:00:00_08:59:59_S0.json
08:00:00_08:59:59_S1.json

These two files together contain the syslog log entries for all instances during the hour beginning 0800 UTC. To get all the log entries, you must read all the shards for each time period—in this case, file shards 0 and 1. The number of file shards written can change for every time period depending on the volume of log entries.

Such high-performance logging services can also offer alternatives to logging to files, management of logfiles can thus be avoided altogether if that is of interest:

• inserting logs directly into a database. For example Stackdriver Logging can push logs directly into Google BigQuery
• pushing logs directly into a processing engine. For example Stackdriver Logging can push logs into Google Pub/Sub topics

Finally - if real-time logfile merging is not a requirement having multiple logfiles can help with offline log management:

• easy to devise progressive log backup, compression, archiving and eventual disposal schemes
• parallel processing of multiple sets of logs (logfiles) is possible, reducing/avoiding bottleneck effects
• no file splitting and re-writing necessary