1

I install a fluentd chart on OKD 3.11 with:

helm install kiwigrid/fluentd-elasticsearch \
--name fluentd \
-f ../../tools/fluentd/values.yaml \
--namespace logging

The pod is created, I can enter in it and I can curl to external elasticsearch successfully. However the liveness probe keeps failing and the pods on timeout get recreated:

Normal   Killing    10m (x3 over 36m) kubelet node1.qa1.isp.ezbob.com 
Killing container with id docker://fluentd-fluentd-elasticsearch:Container failed liveness probe.. Container will be killed and recreated.

And I can't find any indexes being created in ES.

The liveness config is the default for the chart:

 Liveness:       exec [/bin/sh -c LIVENESS_THRESHOLD_SECONDS=${LIVENESS_THRESHOLD_SECONDS:-300}; STUCK_THRESHOLD_SECONDS=${STUCK_THRESHOLD_SECONDS:-900}; if [ ! -e /var/log/fluentd-buffers ]; then
  exit 1;
fi; touch -d "${STUCK_THRESHOLD_SECONDS} seconds ago" /tmp/marker-stuck; if [ -z "$(find /var/log/fluentd-buffers -type d -newer /tmp/marker-stuck -print -quit)" ]; then
  rm -rf /var/log/fluentd-buffers;
  exit 1;
fi; touch -d "${LIVENESS_THRESHOLD_SECONDS} seconds ago" /tmp/marker-liveness; if [ -z "$(find /var/log/fluentd-buffers -type d -newer /tmp/marker-liveness -print -quit)" ]; then
  exit 1;
fi;
] delay=600s timeout=1s period=60s #success=1 #failure=3

The OS is CentOS 7.6. I can't get any info about this issue and no trace to catch, it looks like some permission issues on mounting points.

New contributor
ebdevops1 is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
2

Changing logging driver form journald to json has fixed it. I edited /etc/sysconfig/docker, and set

OPTIONS='--selinux-enabled --log-driver=json-file --signature-verification=false'

Then restarted docker on the nodes I needed:

systemctl daemon-reload && systemctl restart docker
New contributor
ebdevops1 is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
  • 1
    Could you please explain how you changed the logging driver? – Bruce Becker Dec 3 at 12:15
  • vi /etc/sysconfig/docker: set OPTIONS='--selinux-enabled --log-driver=json-file --signature-verification=false' and then systemctl daemon-reload && systemctl restart docker on the nodes I needed. – ebdevops1 Dec 3 at 12:21
  • Great! but could you update the answer with this information? Thanks – Bruce Becker Dec 3 at 12:22

Your Answer

ebdevops1 is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.