I'm currently setting up a stack with Grafana and Grafana Loki to store, query, and analyze logs from different web applications. Most applications run on .NET using Serilog and the Serilog.Sinks.Grafana.Loki-Sink.
From time to time we see errors och warnings in the logs that we know are not relevant or that we want to ignore.
I know there are several options here but none of them are "straightforward":
1. Filtering them in the reporting web app so that they are never sent to Loki
This would require a release of the app just to filter out logs. Does not make sense to me.
2. Filter using LogQL
In this case the data is still stored in Loki and it would also make my queries complicated - would be really hard to make sure all dashboards uses them same filtering logic.
3. Filter using loki-configuration
There seems to be some kind of pre_ingesters-setting in the Loki configuration but using this would probably require a restart of the Loki-service which is not a great option.
I'm basically looking for something where I (as the user) can create a number of filters (preferably using a web UI) that would be applied before Loki store the log lines, like some kind of proxy-layer.
How can this be solved?
