I install a fluentd chart on OKD 3.11 with:

helm install kiwigrid/fluentd-elasticsearch \
--name fluentd \
-f ../../tools/fluentd/values.yaml \
--namespace logging

The pod is created, I can enter in it and I can curl to external elasticsearch successfully. However the liveness probe keeps failing and the pods on timeout get recreated:

Normal   Killing    10m (x3 over 36m) kubelet node1.qa1.isp.ezbob.com 
Killing container with id docker://fluentd-fluentd-elasticsearch:Container failed liveness probe.. Container will be killed and recreated.

And I can't find any indexes being created in ES.

The liveness config is the default for the chart:

 Liveness:       exec [/bin/sh -c LIVENESS_THRESHOLD_SECONDS=${LIVENESS_THRESHOLD_SECONDS:-300}; STUCK_THRESHOLD_SECONDS=${STUCK_THRESHOLD_SECONDS:-900}; if [ ! -e /var/log/fluentd-buffers ]; then
  exit 1;
fi; touch -d "${STUCK_THRESHOLD_SECONDS} seconds ago" /tmp/marker-stuck; if [ -z "$(find /var/log/fluentd-buffers -type d -newer /tmp/marker-stuck -print -quit)" ]; then
  rm -rf /var/log/fluentd-buffers;
  exit 1;
fi; touch -d "${LIVENESS_THRESHOLD_SECONDS} seconds ago" /tmp/marker-liveness; if [ -z "$(find /var/log/fluentd-buffers -type d -newer /tmp/marker-liveness -print -quit)" ]; then
  exit 1;
] delay=600s timeout=1s period=60s #success=1 #failure=3

The OS is CentOS 7.6. I can't get any info about this issue and no trace to catch, it looks like some permission issues on mounting points.

1 Answer 1


Changing logging driver form journald to json has fixed it. I edited /etc/sysconfig/docker, and set

OPTIONS='--selinux-enabled --log-driver=json-file --signature-verification=false'

Then restarted docker on the nodes I needed:

systemctl daemon-reload && systemctl restart docker
  • 1
    Could you please explain how you changed the logging driver? Dec 3, 2019 at 12:15
  • vi /etc/sysconfig/docker: set OPTIONS='--selinux-enabled --log-driver=json-file --signature-verification=false' and then systemctl daemon-reload && systemctl restart docker on the nodes I needed. Dec 3, 2019 at 12:21
  • Great! but could you update the answer with this information? Thanks Dec 3, 2019 at 12:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.