Since two people already advised you to rethink your current process (which I second since it will cause you sleepless nights at some point ;)), I will go another route and recommend a specific piece of software which - in my opinion - fits most of your needs: Graylog.
I implemented and used a couple of ELK stacks for both log aggregation and business intelligence and also run/maintain graylog for about two years now at my current employer. I recommend graylog since it has the following features built-in and is - in my opinion - a bit easier to setup and maintain:
- A web-interface
- Multiuser capabilities
As far as I understand your scenario, it looks as if you need to act or be alerted on certain events that appear in your stream of log messages. If we look at the Graylog features:
Trigger actions or get notified when something needs attention, such as failed login attempts, exceptions or performance degradation.
Ideas: Send an email or Slack message to your team. Spawn a new machine to balance the processing load. Block IP ranges in your firewalls automatically when an attack is detected.
To give graylog a try I'd recommend the following two steps:
- Setup a dedicated host which is reachable by all you application hosts to run graylog (and its dependencies MongoDB and ElasticSearch)
- Send logs from your application to graylog (possibly as GELF messages)
Note: These two steps have the ability to fill pages and pages of best practices and should receive at least a couple of thoughts. Not to mention that graylog isn't a monitoring solution and graylog itself should be monitored by a proper monitoring tool (like e.g. Icinga, Prometheus, Nagios to name just a few).