0

At the moment there are multiple microservices that log to individual files. If there is an issue then the team is debugging like headless chickens. In order to prevent this the aim is to introduce aggregated logging, e.g. elastic. What are best practices to migrate to such an aggregated logging system?

Current idea

What endpoint should be configured?

  • 1
    Is this a "cultural" or a technical question? Did you try what you wrote, did anything go wrong? – AnoE Oct 9 '17 at 5:32
4

One best practice would be to log everything to the same location. Your "Current idea" suggests that you'd be running the Elastic stack locally, which is great for a local development environment, but for anything outside that you want to be shipping the logs from all your microservices to the same location. This enables you to do correlation between logs, and have a "single pane of glass" approach to your logging.

To achieve that, you're going to have to run it on a server or use a managed service such as Elastic Cloud or AWS Elasticsearch. You can install the components through package managers e.g. yum or apt, https://www.elastic.co/start, or it's also possible to run it in Docker. The stack-docker repository is great for running locally, but it's not suitable for any sizeable deployment due to a lack of scalability.

The endpoint you ship to will depend on what method/protocol you're using to ship logs, for example Nginx running on an AWS EC2 instance might use Filebeat and ship to a load balancer that sits in front of Elasticsearch, or if you're using Docker you might use the gelf Docker logdriver to ship your logs to Logstash.

Some more best practices:

  • Log all your messages in JSON, with a focus on objects/fields rather than long string messages.
  • Exceptions should be actionable, anything else can be a warning.
  • Avoid multiline logs.
  • Use correlation IDs. If your app is passed a correlation ID it should include it in every message triggered by that call, otherwise it should generate one. The correlation ID should be included in calls to other microservices.
  • Err on the side of logging more rather than less, you never know when you may need that bit of data, or what trends you might find by analysing an otherwise inconspicuous piece of data.
  • Separate your environments logically so you can have different retention periods and archiving on your logs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.