1

I install a fluentd chart on OKD 3.11 with:

helm install kiwigrid/fluentd-elasticsearch \
--name fluentd \
-f ../../tools/fluentd/values.yaml \
--namespace logging

The pod is created, I can enter in it and I can curl to external elasticsearch successfully. However the liveness probe keeps failing and the pods on timeout get recreated:

Normal   Killing    10m (x3 over 36m) kubelet node1.qa1.isp.ezbob.com 
Killing container with id docker://fluentd-fluentd-elasticsearch:Container failed liveness probe.. Container will be killed and recreated.

And I can't find any indexes being created in ES.

The liveness config is the default for the chart:

 Liveness:       exec [/bin/sh -c LIVENESS_THRESHOLD_SECONDS=${LIVENESS_THRESHOLD_SECONDS:-300}; STUCK_THRESHOLD_SECONDS=${STUCK_THRESHOLD_SECONDS:-900}; if [ ! -e /var/log/fluentd-buffers ]; then
  exit 1;
fi; touch -d "${STUCK_THRESHOLD_SECONDS} seconds ago" /tmp/marker-stuck; if [ -z "$(find /var/log/fluentd-buffers -type d -newer /tmp/marker-stuck -print -quit)" ]; then
  rm -rf /var/log/fluentd-buffers;
  exit 1;
fi; touch -d "${LIVENESS_THRESHOLD_SECONDS} seconds ago" /tmp/marker-liveness; if [ -z "$(find /var/log/fluentd-buffers -type d -newer /tmp/marker-liveness -print -quit)" ]; then
  exit 1;
fi;
] delay=600s timeout=1s period=60s #success=1 #failure=3

The OS is CentOS 7.6. I can't get any info about this issue and no trace to catch, it looks like some permission issues on mounting points.

2

Changing logging driver form journald to json has fixed it. I edited /etc/sysconfig/docker, and set

OPTIONS='--selinux-enabled --log-driver=json-file --signature-verification=false'

Then restarted docker on the nodes I needed:

systemctl daemon-reload && systemctl restart docker
| improve this answer | |
  • 1
    Could you please explain how you changed the logging driver? – Bruce Becker Dec 3 '19 at 12:15
  • vi /etc/sysconfig/docker: set OPTIONS='--selinux-enabled --log-driver=json-file --signature-verification=false' and then systemctl daemon-reload && systemctl restart docker on the nodes I needed. – anVzdGFub3RoZXJodW1hbg Dec 3 '19 at 12:21
  • Great! but could you update the answer with this information? Thanks – Bruce Becker Dec 3 '19 at 12:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.