2

Is it possible to have several brokers behind a single TCP/IP port?

I.e., offer a single endpoint for Kafka clients, and having a scalable amount of brokers behind that?

In my company, some people are configuring a (scaling) Kafka server for us, and they insist that this is not possible, and that we need to configure every single broker as a separate endpoint...

This is in a OpenShift context; it would be nice to just be able to scale the Kafka pods up and down as needed, while using the usual loadbalancing to offer a single port to the outside world.

EDIT: I am familiar with HAProxy; the question is about Kafka, not OpenShift/ HAProxy. Is there anything special to be configured (in the Kafka broker image)

  • Are they all synchronized amongst each other?
  • Are there examples for that somewhere?

Googling for "OpenShift Apache Kafka" etc. brings up plenty of examples, but all that I found seem to have a different service for each Apache Kafka broker (i.e., explicitly running 3 separate Kafka deployments inside OpenShift, with separate OpenShift services, one each). This is what I wish to avoid. I want to horizontally scale the number of Kafka processes behind HAProxy, and have one single external IP.

Improve this question

1 Answer 1

Reset to default
1

Yes, you can. HAProxy is purpose built to balance any TCP or HTTP connection. My old company used to run every component of Kafka with HAproxy.

Here is a whole project dedicated to running auto-scaling Kafka in Docker using HAproxy: https://github.com/pozgo/docker-kafka

You can even see their HAproxy config here where they specify the broker check (port 9092): https://github.com/pozgo/docker-kafka/blob/master/haproxy.cfg

Improve this answer
2
  • Thank you @BoomShadow... I am pretty familiar with HAProxy. See my edit for a bit of clarification. I am strongly pondering to delete the question because I'm not happy with it - I think I need to learn more about Kafka to express what I really mean (basically, how to proceed when the pod of a single broker is scaled up, how they then manage to synchronize their data - both reading and writing, and so on). But meanwhile, your answer might be useful for others.
    – AnoE
    Commented Feb 23, 2018 at 16:56
  • Ah, I gotcha. Your question would actually be more of "how to do" rather than "can one do". I wish I could help more there. I don't have access to the custom config files used by my old team, or else I would share snippets here with you. I personally hadn't touched that part of the infrastructure; I just knew it was there and it's basic arrangement. I think you should leave your question up and see if anyone else can jump in and offer solid insight of what you're looking for. The edit you made clarified things well.
    – BoomShadow
    Commented Feb 23, 2018 at 19:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.