One best practice would be to log everything to the same location. Your "Current idea" suggests that you'd be running the Elastic stack locally, which is great for a local development environment, but for anything outside that you want to be shipping the logs from all your microservices to the same location. This enables you to do correlation between logs, and have a "single pane of glass" approach to your logging.
To achieve that, you're going to have to run it on a server or use a managed service such as Elastic Cloud or AWS Elasticsearch. You can install the components through package managers e.g. yum or apt, https://www.elastic.co/start, or it's also possible to run it in Docker. The stack-docker repository is great for running locally, but it's not suitable for any sizeable deployment due to a lack of scalability.
The endpoint you ship to will depend on what method/protocol you're using to ship logs, for example Nginx running on an AWS EC2 instance might use Filebeat and ship to a load balancer that sits in front of Elasticsearch, or if you're using Docker you might use the gelf Docker logdriver to ship your logs to Logstash.
Some more best practices:
- Log all your messages in JSON, with a focus on objects/fields rather than long string messages.
- Exceptions should be actionable, anything else can be a warning.
- Avoid multiline logs.
- Use correlation IDs. If your app is passed a correlation ID it should include it in every message triggered by that call, otherwise it should generate one. The correlation ID should be included in calls to other microservices.
- Err on the side of logging more rather than less, you never know when you may need that bit of data, or what trends you might find by analysing an otherwise inconspicuous piece of data.
- Separate your environments logically so you can have different retention periods and archiving on your logs.
