0

I have an application which processes a single stream of data. Each stream has a unique identifier and a URL. The number of streams is dynamic, but I only want one instance of the application to process each stream.

I want to build the infrastructure to have a single application to process each stream, and expand dynamically when new streams are added. (You can assume that the list of streams is in a database) If the processor of the stream dies, it needs to be restarted automatically.

I was exploring using Kubernetes StatefulSets for this, since each instance will have its unique identifier, but I don't know how to link these to a unique stream, and how to trigger new instances to be executed when new streams are added (as automatically as possible). Maybe using different namespaces, but I want to be able to do it dynamically.

I am a bit new to Kubernetes, so not too sure if I am missing something obvious. Is there a way to achieve this? Similar to how for example databases have master and slaves with different configurations, but more dynamic since streams can be added or deleted during execution.

How is it best to go about something like this?

Improve this question

1 Answer 1

Reset to default
0

KEDA can best suit this scenario. Event based scaling can help autoscale the K8s pods accordingly based on the chosen events. KEDA offers mutiple event options to act as scaling factor. Reference: https://keda.sh/docs/2.0/scalers If no available event triggers can be used, an external scaler can also be used as an API which can be used as HPA for Kuberenetes. https://keda.sh/docs/2.0/scalers/external/

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.