I have a multinode kubernetes cluster. Multiple services are deployed as Pods. They communicate over each other via rabbitmq which also exists as Pod in the Cluster.

Problem Scenario:

Many time services fails to connect to required queue in the Rabbitmq. Log for the same are reported in Rabbitmq pod logs and on the services Pod as well. This occurs primarily due to connectivity issues and is inconsistent. Due to this failure functionality breaks. And also since this is NOT a crash, pod is always in running state in the kubernetes. To fix this we have to manually go and restart the pod.

I want to create a liveness probe for every pod. But how this should work to catch the exception? Since many process in a service can be trying to access the connection, any one of them can fail.

  • If you ask rabbitmq its status during these "connectivity" issues, would you get a positive or a negative health? – gumol Aug 19 '18 at 21:44
  • Also, which pod you restart? Rabbit or the consumers? – gumol Aug 19 '18 at 21:45
  • Rabbitmq is up and running k8 environment, yet its error logs mentions "failed to authenticate queues" of that service pod. I believe once there is a drop in connection rabbitmq is not able to figure out if it has to reinitalize the queue with the service. It thinks queue is active and exclusivley consumed by the service. And possible because of the same reason when same service tries to access the same queue, it fails to authorize queue. I want to restart the service pod and reinitialize the service-queues. – C.v Aug 20 '18 at 5:58
  • Ok, I see and by re-initializing the service queues, you mean deleting the queues, purging them or restart rabbit? – gumol Aug 20 '18 at 11:47
  • mostly by restarting the consumer pod, consumer service is able to connect rabbit properly. Same also when i restart the rabbit pod. But i usually avoid restarting rabbit. – C.v Aug 20 '18 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.