I am planning to create RabbitMQ cluster using Ansible on AWS VPC with Amazon internal load balancer as the frontend to point connections to it.

Any suggestion how to remove a dead node from RabbitMQ cluster based on autoscaling rule where nodes can go up and down, or if you are using spot instances?

When a node goes down, RabbitMQ does not remove it from the replication list automatically, I can see Node not running in the management UI.

I managed to join to the cluster a scaled instance automatically via Ansible and userdata.

Diagram of Infrastructure

  • @Pierre.Vriens, I changed to only 1 question, thanks. – Berlin Mar 25 '17 at 15:33
  • Merci! You didn't destroy the other 4 (or so) questions I hope. Maybe keep them as folowup question, to the extend they are still relevant? – Pierre.Vriens Mar 25 '17 at 15:35
  • Yep, the questions are still relevant, but this one is the most important. I will post another question later :) thanks! – Berlin Mar 25 '17 at 15:39
  • 1
    @Berlin I drew a diagram that represents what I think you are describing, if you meant something else then please do let me know and I will adapt. – Richard Slater Mar 25 '17 at 18:59
  • 1
    Hey @Pierre.Vriens - I don't mind I had a few minutes and wanted to clarify my assumption, theoretically, I could add it to my answer, and I may well do that. – Richard Slater Mar 25 '17 at 19:45

Consider using the rabbitmq/rabbitmq-autocluster plugin:

A RabbitMQ plugin that clusters nodes automatically using a number of peer discovery mechanisms:

  • Consul,
  • etcd2
  • DNS A records
  • AWS EC2 tags
  • AWS Autoscaling Groups

There is a fair bit of configuration to plug in to get this setup including setting IAM policies and adding EC2 tags to the instances you want to be party to your cluster.

If you were to use AWS Autoscaling Groups then you would add the following to your rabbitmq.config:

  {rabbit, [ ... ]},
  {autocluster, [
    {backend, aws},
    {aws_autoscaling, true},
    {aws_ec2_region, "us-west-2"}

If you are not using AWS Autoscaling Groups you can still achieve the desired result using tags on your EC2 Instances:

  {rabbit, [ ... ]},
  {autocluster, [
    {backend, aws},
    {aws_ec2_tags, [{"region", "us-west-2"}, {"service", "rabbitmq"}]},
    {aws_ec2_region, "us-east-1"},
    {aws_access_key, "..."},
    {aws_secret_key, "..."}

With all of that said I strongly recommend using Consul by HashiCorp as your service discovery mechanism, in the long run, you get significantly more flexibility in terms of decoupling your parts of your system from each other.

  • thank you for the detailed explanation and the diagram, so AFAIK rabbitmq/rabbitmq-autocluster plugin will know also to remove node from the replication list once the node is down, one more thing if may I ask, I thought to start with 2-node cluster, are you suggest to start with 3-node cluster as you describe in your diagram with policy ` rabbitmqctl set_policy ha-all "" '{"ha-mode":"all","ha-sync-mode":"automatic"}' ` ? or should I post it in a another question ? – Berlin Mar 25 '17 at 22:06
  • 1
    Possibly another question as the formatting in comments isn't all that easy to use. I have always advised clients when adopting cloud to use larger number of small instances rather than the inverse. This is because failure is expected in the cloud, if you have a 2-node cluster and 1 fails you have lost 50% of your capacity, whereas if you have a 3-node cluster and 1 fails you only lose 33%. – Richard Slater Mar 26 '17 at 8:47
  • I have configured my cluster using rabbitmq/rabbitmq-autocluster plugin and it is work pretty well however when node goes down RabbitMQ does not remove it from the replication list, any idea why ? – Berlin Apr 1 '17 at 15:13
  • 1
    Have you enabled the Cluster Cleanup configuration option? – Richard Slater Apr 1 '17 at 16:18
  • thanks, found it https://github.com/aweber/rabbitmq-autocluster/wiki/General-Settings, I will try that. – Berlin Apr 1 '17 at 22:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.