Will Docker Swarm work if masters are spread over 3 continents?

Question

I need to deploy a highly available Docker Swarm on Azure. I created a VNet in 3 locations: Brazil South (my location), East US and West Europe. Then I enabled network peering so VMs in these locations can see each other as if they were in the same VNet. Cluster seems working fine. The problem is that turns out latency is high between these locations: ~190ms.

Questions:

1. Is this a recommended topology for HA? Is there a way to improve it?
2. Will this latency pose any (future) limitations?

Answer 1

There are various timeouts within swarm mode that do not handle high latency well and may start to assume nodes within the cluster are down, lose quorum, or have networking issues. For HA, the standard advice is to have multi-AZ but not multi-region for a single cluster. For use cases that need multi-region, you would deploy multiple swarm clusters, one per region. If you really need multi-region with a single swarm cluster, then consider placing all of the management nodes within the same region and only deploying worker nodes to the other regions.