DevOps Stack Exchange is a question and answer site for software engineers working on automated testing, continuous delivery, service integration and monitoring, and building SDLC infrastructure. It only takes a minute to sign up.
Sign up to join this community
if i have filebeat -> logstash -> elasticsearch pipeline with TLS or HTTPS encryption , is it possible to load balance between 2 destination elasticsearch clusters in active-standby ?
what i mean is, in case one of the clusters goes down i would manually or automatically switch all the data streams to the other cluster.
im looking at various proxy setups but im not sure i understand the terms correctly, and TLS would not solve the problem.
You can do this a couple of ways, but I have never seen anyone do this with a CNAME.
One option, if you have control over your IP addressing is to use BGP to fail-over (or load balance) an IP to your cloud hosting providers.
A second option is to invest in a DNS-based geographic load balancing solution, such as the GTM from F5, the GSLB solution from A10, or the GSLB solution from Kemp (F5 is the most commonly used in the industry). With these solutions you will set your name server (NS1 and NS2) for your domain to be the GSLB solution of your choice. You will set a very low TTL for your DNS records and configure health monitors to your ELK stack from the GSLB solution. These can be as complex as logging into ELK and scraping for data or as simple as making sure you can establish a TCP connection and/or ping your server. When this health monitor fails, the GSLB solution will failover traffic to the alternate data center automagically. You will be able to do this as an HA pair with one GSLB solution at once DC as NS1 and the other at your secondary site as NS2. The GSLB will then forward lookup to your previous nameservers which will publish IP addresses for both sites. These will be rewritten to eliminate the IP that the GSLB does not want to direct traffic to at the time (again, this can be used for active-active sites - you can even use geolocation to send users in Europe to a Eurpoean cloud provider and US based users to a US based provider, or do a simple active/standby)
A final way to do this might be with local load balancing solution, such as F5's LTM, A10's SLB, or Kemp's solution. You may also be able to use nginx for this for a cheap solution. In this scenario, you might have this solution onsite in an office with the URL to the load balancer served up by your local DNS server to a local IP (a Virtual IP) on your load balancer. You would then configure the load balance to direct users to the desired cloud provider by configuring a health monitor (of whatever complexity). You would then need to configure this load balancer to bridge your SSL connection. You would add your certificate and private key to the load balancing solution. Users would establish an encrypted connection to the load balancer. This encryption would be stripped off by the load balancer and a new encrypted connection established to the cloud provider by the load balancer, and the data forwarded along over the new SSL tunnel.
