Background: When running Consul client-mode in AWS, EC2s are constantly getting replaced, which results in a lot of dead client nodes when looking at the Consul UI or via consul members.

Already tried to enabled reconnect_timeout, leave_on_terminate (default to true on Client but tried to turn on anyway), and Autopilot cleanup_dead_servers to no avail.

The delay seems to be inconsistent as well, some got reaped after 4~7 days, some are never reaped, consul force-leave had to be used.

  • which consul version you are using?
    – Samit
    Commented Nov 20, 2019 at 19:00
  • Both servers and clients are on 1.0.0
    – Casper
    Commented Nov 20, 2019 at 21:51

1 Answer 1


As a best practice, you should gracefully deregister the node gracefully. In this case, consul will know the the node has left and will be removed from the cluster. Otherwise, the consul cannot distinguish between a temporary failure, agent crash, network partition, etc.

There is a github issue as well related to this.

Quoting few important points from the issue.

The nodes should automatically reap out after 72hours (not yet configurable, but soon). Otherwise, the best route is to issue a graceful leave before destroying the nodes (consul leave), so that they can be reaped immediately. They are kept around for that long since without a graceful leave, Consul cannot distinguish between a temporary failure, agent crash, network partition, etc.


force-leave should push them into the "left" state. Nodes are not reaped until they are in failed for 72h or in the left state. "force-leave" just moves a node from the "failed" -> "left" state. They are not removed from the members list for 24 or 72h.

  • What I'm trying to figure out is that is there a configurable setting for reap time on the clients (for now only Consul servers have these settings). Gracefully leave is no longer sustainable for cloud environment anymore since VMs are very redundant and nobody cares whether a node is shutdown.
    – Casper
    Commented Nov 20, 2019 at 21:57
  • 2
    As of now, this property is not available for agents. There is a github issue for enabling this. Please refer github.com/hashicorp/consul/issues/2982
    – Samit
    Commented Nov 21, 2019 at 14:28
  • had to do the ugly force-leave command also to remove federation between 2 consul (VM - K8s) | also doing the -prune will give me "the flag provided but not defined: -prune" | consul version 1.1.0
    – Madox
    Commented Sep 1, 2021 at 13:10
  • @Casper Would you be able to accept this answer if this had fixed the issue? This will help other people.
    – Samit
    Commented Jan 15, 2023 at 16:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.