Is it possible to have several brokers behind a single TCP/IP port?
I.e., offer a single endpoint for Kafka clients, and having a scalable amount of brokers behind that?
In my company, some people are configuring a (scaling) Kafka server for us, and they insist that this is not possible, and that we need to configure every single broker as a separate endpoint...
This is in a OpenShift context; it would be nice to just be able to scale the Kafka pods up and down as needed, while using the usual loadbalancing to offer a single port to the outside world.
EDIT: I am familiar with HAProxy; the question is about Kafka, not OpenShift/ HAProxy. Is there anything special to be configured (in the Kafka broker image)
- Are they all synchronized amongst each other?
- Are there examples for that somewhere?
Googling for "OpenShift Apache Kafka" etc. brings up plenty of examples, but all that I found seem to have a different service for each Apache Kafka broker (i.e., explicitly running 3 separate Kafka deployments inside OpenShift, with separate OpenShift services, one each). This is what I wish to avoid. I want to horizontally scale the number of Kafka processes behind HAProxy, and have one single external IP.