We have a practical question I struggle to answer :
What is a good logfile schema?
Here are the caveats:
- There are several workloads, each writing more than one log file
- Applications are on more than one language
- All applications are in the cloud, but on different cloud providers
- All logs are sent to a central dashboard (e.g. Kibana) which can be queried.
The requirements which this schema should fulfill are:
- must be queryable (structured)
- should be able to determine failure patterns (correlations)
- no proprietary file formats
For a particular case, we may want to set a trigger of some action based on a particular query.
Assuming, e.g. we write logs in JSON format, what would be a good schema to apply to the entries, to satisfy the caveats and requirements?