Background: When running Consul client-mode in AWS, EC2s are constantly getting replaced, which results in a lot of dead client nodes when looking at the Consul UI or via consul members.

Already tried to enabled reconnect_timeout, leave_on_terminate (default to true on Client but tried to turn on anyway), and Autopilot cleanup_dead_servers to no avail.

The delay seems to be inconsistent as well, some got reaped after 4~7 days, some are never reaped, consul force-leave had to be used.

  • which consul version you are using?
    – Samit
    Nov 20 '19 at 19:00
  • Both servers and clients are on 1.0.0
    – Casper
    Nov 20 '19 at 21:51

As a best practice, you should gracefully deregister the node gracefully. In this case, consul will know the the node has left and will be removed from the cluster. Otherwise, the consul cannot distinguish between a temporary failure, agent crash, network partition, etc.

There is a github issue as well related to this.

Quoting few important points from the issue.

The nodes should automatically reap out after 72hours (not yet configurable, but soon). Otherwise, the best route is to issue a graceful leave before destroying the nodes (consul leave), so that they can be reaped immediately. They are kept around for that long since without a graceful leave, Consul cannot distinguish between a temporary failure, agent crash, network partition, etc.


force-leave should push them into the "left" state. Nodes are not reaped until they are in failed for 72h or in the left state. "force-leave" just moves a node from the "failed" -> "left" state. They are not removed from the members list for 24 or 72h.

  • What I'm trying to figure out is that is there a configurable setting for reap time on the clients (for now only Consul servers have these settings). Gracefully leave is no longer sustainable for cloud environment anymore since VMs are very redundant and nobody cares whether a node is shutdown.
    – Casper
    Nov 20 '19 at 21:57
  • 2
    As of now, this property is not available for agents. There is a github issue for enabling this. Please refer github.com/hashicorp/consul/issues/2982
    – Samit
    Nov 21 '19 at 14:28
  • had to do the ugly force-leave command also to remove federation between 2 consul (VM - K8s) | also doing the -prune will give me "the flag provided but not defined: -prune" | consul version 1.1.0
    – Madox
    Sep 1 at 13:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.