This answer is more about scalability considerations - if the number of workers can be high and/or multiple of them can produce logs at high rate at the same time.
Yes, using multiple logfiles simultaneously is a good practice.
Attempting to combine into a single logfile logs from multiple workers in real time will raise problems:
- using blocking mechanims to prevent message loss will slow down the workers
- log messages can appear out-of-order in the combined logfile
- a centralized logging facility which combines the logs can be overloaded due to limited write speed, messages would be lost
Sharding logfiles (using multiple logfiles active in the same time) is itself a technique used by some hosting providers offering high performance, scalable centralized logging services. For example, when exporting logs to files Google's StackDriver Logging produces multiple sharded logfiles. From Log entries in Google Cloud Storage:
When you export logs to a Cloud Storage bucket, Stackdriver
Logging writes a set of files to the bucket. The files are organized
in directory hierarchies by log type and date. The log type can be a
simple name like syslog
or a compound name like
appengine.googleapis.com/request_log
. If these logs were stored in a
bucket named my-gcs-bucket
, then the directories would be named as
in the following example:
my-gcs-bucket/syslog/YYYY/MM/DD/
my-gcs-bucket/appengine.googleapis.com/request_log/YYYY/MM/DD/
A single bucket can contain logs from multiple log types.
The leaf directories (DD/
) contain multiple files, each of which
holds the exported log entries for a time period specified in the file
name. The files are sharded and their names end in a shard number,
Sn
or An
(n=0, 1, 2, ...). For example, here are two files that
might be stored within the directory
my-gcs-bucket/syslog/2015/01/13/
:
08:00:00_08:59:59_S0.json
08:00:00_08:59:59_S1.json
These two files together contain the syslog
log entries for all
instances during the hour beginning 0800 UTC. To get all the log
entries, you must read all the shards for each time period—in this
case, file shards 0 and 1. The number of file shards written can
change for every time period depending on the volume of log entries.
Such high-performance logging services can also offer alternatives to logging to files, management of logfiles can thus be avoided altogether if that is of interest:
Finally - if real-time logfile merging is not a requirement having multiple logfiles can help with offline log management:
- easy to devise progressive log backup, compression, archiving and eventual disposal schemes
- parallel processing of multiple sets of logs (logfiles) is possible, reducing/avoiding bottleneck effects
- no file splitting and re-writing necessary