How to automatically remove a dead node from RabbitMQ cluster

Question

I am planning to create RabbitMQ cluster using Ansible on AWS VPC with Amazon internal load balancer as the frontend to point connections to it.

Any suggestion how to remove a dead node from RabbitMQ cluster based on autoscaling rule where nodes can go up and down, or if you are using spot instances?

When a node goes down, RabbitMQ does not remove it from the replication list automatically, I can see Node not running in the management UI.

I managed to join to the cluster a scaled instance automatically via Ansible and userdata.

Answer 1

Consider using the rabbitmq/rabbitmq-autocluster plugin:

A RabbitMQ plugin that clusters nodes automatically using a number of peer discovery mechanisms:

• Consul,
• etcd2
• DNS A records
• AWS EC2 tags
• AWS Autoscaling Groups

There is a fair bit of configuration to plug in to get this setup including setting IAM policies and adding EC2 tags to the instances you want to be party to your cluster.

If you were to use AWS Autoscaling Groups then you would add the following to your rabbitmq.config:

[
  {rabbit, [ ... ]},
  {autocluster, [
    {backend, aws},
    {aws_autoscaling, true},
    {aws_ec2_region, "us-west-2"}
  ]}
].

If you are not using AWS Autoscaling Groups you can still achieve the desired result using tags on your EC2 Instances:

[
  {rabbit, [ ... ]},
  {autocluster, [
    {backend, aws},
    {aws_ec2_tags, [{"region", "us-west-2"}, {"service", "rabbitmq"}]},
    {aws_ec2_region, "us-east-1"},
    {aws_access_key, "..."},
    {aws_secret_key, "..."}
  ]}
].

With all of that said I strongly recommend using Consul by HashiCorp as your service discovery mechanism, in the long run, you get significantly more flexibility in terms of decoupling your parts of your system from each other.