0

I'm having some troubles with setting up ElasticSearch with Fluentd and Fluentbit for log aggregation.

I'm using the following chart: https://github.com/helm/charts/tree/master/stable/elastic-stack

ElasticSearch is running and all indices and nodes are healthy.

Previously, I was using the fluentd-elasticsearch plugin, and this worked when sending directly to elasticsearch.

I'm now trying to use Fluentbit for collecting, Fluentd for forwarding but this is having problems delivering the forwarded data from Fluentd -> ElasticSearch.

The forwarding from FluentBit -> Fluentd is fine.

I'm seeing the following exception in the Fluentd pod logs:

2019-05-21 08:57:03 +0000 [warn]: #0 [elasticsearch] failed to flush the buffer. retry_time=1 next_retry_seconds=2019-05-21 08:57:04 +0000 chunk="5896207ac8a9863d02e19a5b261af84f" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elastic-elasticsearch-client\", :port=>9200, :scheme=>\"http\"}): connect_write timeout reached"
  2019-05-21 08:57:03 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-3.0.2/lib/fluent/plugin/out_elasticsearch.rb:655:in `rescue in send_bulk'
  2019-05-21 08:57:03 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-3.0.2/lib/fluent/plugin/out_elasticsearch.rb:638:in `send_bulk'
  2019-05-21 08:57:03 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-3.0.2/lib/fluent/plugin/out_elasticsearch.rb:544:in `block in write'
  2019-05-21 08:57:03 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-3.0.2/lib/fluent/plugin/out_elasticsearch.rb:543:in `each'
  2019-05-21 08:57:03 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-3.0.2/lib/fluent/plugin/out_elasticsearch.rb:543:in `write'
  2019-05-21 08:57:03 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.3.3/lib/fluent/plugin/output.rb:1123:in `try_flush'
  2019-05-21 08:57:03 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.3.3/lib/fluent/plugin/output.rb:1423:in `flush_thread_run'
  2019-05-21 08:57:03 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.3.3/lib/fluent/plugin/output.rb:452:in `block (2 levels) in start'
  2019-05-21 08:57:03 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.3.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2019-05-21 08:57:09 +0000 [warn]: #0 [elasticsearch] failed to flush the buffer. retry_time=2 next_retry_seconds=2019-05-21 08:57:10 +0000 chunk="5896207ac8a9863d02e19a5b261af84f" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elastic-elasticsearch-client\", :port=>9200, :scheme=>\"http\"}): connect_write timeout reached"

I've tried increasing buffer limits, but this then throws a different problem for every log entry:

2019-05-21 08:41:51 +0000 [warn]: #0 [elasticsearch] failed to flush the buffer. retry_time=81 next_retry_seconds=2019-05-21 08:42:22 +0000 chunk="589616a12c439e0f51f536b487cf4bf5" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elastic-elasticsearch-client\", :port=>9200, :scheme=>\"http\"}): read timeout reached"
  2019-05-21 08:41:51 +0000 [warn]: #0 suppressed same stacktrace
2019-05-21 08:41:51 +0000 [warn]: #0 [elasticsearch] failed to flush the buffer. retry_time=82 next_retry_seconds=2019-05-21 08:42:23 +0000 chunk="589616aca051b55fb2272de7419512bb" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elastic-elasticsearch-client\", :port=>9200, :scheme=>\"http\"}): read timeout reached"

I've tried increasing the clients, but this hasn't changed anything.

Does anyone know what might be the cause of this?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.