This is a micro services deployment question. How would you deploy Envoy SDS(service discovery service) so other envoy proxies can find the SDS server hosts, in order to discover other services to build the service mesh. Should I put it behind a load balancer with a DNS name( single point of failure) or just run the SDS locally in every machine so other micro services can access it? Or is there a better way of deployment that SDS cluster can be dynamically added into envoy config without a single point of failure?
Putting it behind a DNS name with a load balancer across multiple SDS servers is a good setup for reasonable availability. If SDS is down, Envoy will simple not get updated, so it's generally not the most critical failure -- new hosts and services simply won't get added to the cluster/endpoint model in Envoy.
If you want higher availability, you set up multiple clusters. If you add multiple entries to your bootstrap config, Envoy will fail over between them. You can either specify multiple DNS names or multiple IPs.
I ended up listing the SDS/EDS directly under the private DNS zone. However this means it is required to have a static port that the SDS listens to on any box its placed, and every other service will have to have it passed as a known constant. So backend micro services will get a set of ips for available SDS servers, in order to retrieve app service endpoint information. This does not include a load balancer. It can use DNS supported load balancing features. SDS uses a script to heartbeat itself to the DNS. It was implemented in AWS so SDS is running in ECS containers, automatically placed by ECS cluster management service, and that heartbeat script in the SDS server uses AWS metadata service to obtain and report its ip address to DNS. This method means there is very less downtime, since DNS servers hardly ever fail. But we dont get advanced load balancing. Also better than having an SDS in each ec2 host, since it will cost less resources, and will not run into the possibility of all micro services in a particular box failing since its local sds server failed.