So still not benchmarked much, but after a quick code inspection and a bunch of reading about netty and logstash in version 5 the input is not the bottleneck to worry about.
Logstash team did put a bunch of work in the way the filters and outputs plugins are run in parallel, the beats input plugin wait for a batch of events, and the performances problem ...
Both are common and comes with their own pro and cons.
It depends if you can accept loosing logs if logstash is dead and your app restart and on the other hand if you can write logs to disk and consume place there.
Those are the basic points to choose for one or the other, you may end up with more caveats in term of IO/caching/etc.
There's no 'best' ...
After looking a bit further, I found it in the sample called filebeat.full.yml
So I added this section to the filebeat.yml
After running at 100% CPU for minute, Kibana is now immediately showing the results. That was easier than I expected.
#========================== Modules configuration ============================
You might be able to use Filebeat directly. Newer versions of filebeat has a concept of "modules", which is custom parsers you can apply. There are built-in ones for stuff like nginx, apache2, redis etc. Not sure how one goes about writing custom modules, but it might be an avenue worth exploring.
That said, more advanced/custom parsing is likely better ...
I am not sure if you are open to commercial solutions, but Sumo Logic is an excellent ingest-type solution. I also tried Logly and LogDNA which were nice but not as powerful and feature rich as Sumo. Let me also say am not a shill for this company. I’m just a DevOps guy who hates dealing with crappy and incomplete log collection.
After dealing with the ...
"I have two Filebeat pipes inputting into Logstash."
---I assume this means that you have conditional logic and/or prospectors in your filebeat config to ship to the multiple logstash (5044 and 5043)ports? I'd be interested to see that config.
Here's another approach that might acheive your goal: Use the source value in a conditional output. An example of ...
There are two ways to accomplish this, though one of them was only available recently.
The old-school version, the one you can do as far back as Logstash 1.5, is to pay attention to tags and use conditionals to separate your inputs. Roughly...
port => 1525
codec => json_lines
tags => [ 'tcp' ]
ELK is pretty ineffective as a monitoring tool in a default configuration as it doesn't do monitoring - only reporting. ELK can only provide monitoring and alerting abilities if you configure additional add-ons such as ElastAlert.
Then, you are impacted by your ability to log errors via syslog. If messages come in via, say, SNMP traps, you will be unable to ...
You can do this a couple of ways, but I have never seen anyone do this with a CNAME.
One option, if you have control over your IP addressing is to use BGP to fail-over (or load balance) an IP to your cloud hosting providers.
A second option is to invest in a DNS-based geographic load balancing solution, such as the GTM from F5, the GSLB solution from A10, ...
Being unfamiliar with Kibana, I was not aware of the time constraint on data on the default search/display of just 15 minutes. The data coming in was timestamped (@timestamp key) via the 'date' plugin with the original open date, NOT the time of the, via TCP port to elastic, insertion event; thus, no data was showing and I had no idea that by default only ...