I'm fairly new to filebeat, ingest, pipelines in ElasticSearch and not sure how they relate.
In my old environments we had ELK with some custom grok patterns in a directory on the logstash-shipper to parse java stacktraces properly. The logstash indexer would later put the logs in ES.
How do I do this without Logstash? And how do I parse custom log formats?
You might be able to use Filebeat directly. Newer versions of filebeat has a concept of "modules", which is custom parsers you can apply. There are built-in ones for stuff like nginx, apache2, redis etc. Not sure how one goes about writing custom modules, but it might be an avenue worth exploring.
That said, more advanced/custom parsing is likely better left to Logstash than any other component in the Elastic stack.
I am not sure if you are open to commercial solutions, but Sumo Logic is an excellent ingest-type solution. I also tried Logly and LogDNA which were nice but not as powerful and feature rich as Sumo. Let me also say am not a shill for this company. I’m just a DevOps guy who hates dealing with crappy and incomplete log collection.
After dealing with the complexity of ELK, which is admittedly is very powerful and flexible so I have read, ElasticSearch is not for the faint of heart, especially when you walk into a shop and their ELK stack is 4 years old. It’s important to know right right off that Sumo is a cloud solution, so as long as you’re okay with shipping your logs and metrics over HTTPS to their servers, it’s a good way to go. Then you can focus on just feature configuration without having to maintain the core service. It offers most of what I remember that Splunk does without being prohibitively expensive. As you might have guessed, I just completed a project where I moved from ELK to Sumo for production log data and metrics visualization and it is a world of difference from Kibana as far as how much easier it is to get useful output. I found statistical capabilities similar as well, but Kibana does seem to be more actuarially adherent. It’s easy to set up alarms, integrate with RESTful APIs for tools like like Slack, perform complex data correlation, and make purdy dashboards for management. It’s comprehensive as it is an entire stack. Also, there is a very well-developed chef cookbook so installation and configuration is very manageable once you get your ingestion configured along with rulesets. That can be time consuming but as far as cookbooks go I’d say it’s medium complexity. Data ingestion is incredibly flexible- I have not hit any limitations with what it can do, but I did give up trying to ingest directly from logstash as getting metadata to map was more trouble than it was worth. While slurping text logs is what they are geared for, pushing logs and metrics is also an option. We continue to use logback for our app logging, but now Sumo collects pretty much all our logs now as well as graphite data. It does Windows Event Logs nicely as well. We are starting to integrate fluentd from k8s but we’re having some challenges keeping the log volume tamed.
They have their own pre-built support setups for a large amount of OSs, app frameworks, and infrastructure elements which can make initial setup that much faster, but some vendors like Fastly even have advanced prebuilt Sumo Logic implementations that are downright fantastic so our CDN logs create a remarkable dashboard giving an excellent picture of site health, load, as well tons of other useful data. There is definitely a learning curve but nothing like ELK. Getting all your regex just right so things like exceptions in app logs are handled properly can take time, but you can get a lot going with very little effort as a lot of tedious problems like automatic timestamp detection and multi line detection are built in if you want them. Ingestion can be as sophisticated as you want it to be. I found set up kind of evolutionary over the last year where I started with a very basic setup where I did all my pattern matching in my queries, but eventually I had ingestion rules that sifted all my fields out for me which made things more user-friendly for other users. Also, support and documentation are very good. As far as paid services go, this one seemed quite worth it unless you’re already and ace with ELK.
Boy, that was probably a lot more info than you bargained for, but it’s a fair breakdown of Sumo Logic. (I hope!)
