What considerations should be taken into account for a good log file format?

Question

We have a practical question I struggle to answer :

What is a good logfile schema?

Here are the caveats:

• There are several workloads, each writing more than one log file
• Applications are on more than one language
• All applications are in the cloud, but on different cloud providers
• All logs are sent to a central dashboard (e.g. Kibana) which can be queried.

The requirements which this schema should fulfill are:

• must be queryable (structured)
• should be able to determine failure patterns (correlations)
• no proprietary file formats

For a particular case, we may want to set a trigger of some action based on a particular query.

Assuming, e.g. we write logs in JSON format, what would be a good schema to apply to the entries, to satisfy the caveats and requirements?

Answer 1

If you are dealing with application logs, here are the three broad categories of information that a good log file should cover:

• basic context (timestamp, log level, application name, source file name, source module/function, source line number, log message)
• server context (host name, data center name unless host name has it encoded, cluster/pod info, container info)
• user context (request URI, username or a hash, request ID etc)

If we are using JSON for logging, care must be taken to make sure that the keys are short enough to reduce the redundant overhead and also make it easier to query. Using all lowercase and underscores for keys could be a better idea because some of the log aggregators like Splunk are case sensitive.

In the structured format, stack traces become a little tricky to handle. One solution is to use a marker line to indicate the end of a long line rather than just the newline.