I have a service that uses the Kubernetes API to watch the execution of some pods in my cluster. I want multiple replicas of this service running at the same time, to increase resilience.
However, if each of my service's replicas create a watch on a resource, they will all get notified on state changes and they will all act on it.
How can I make only one replica handle each event?
Edit: Someone on the Kubernetes Slack told me, that there is no solution for this built into Kubernetes right now.
I got around my issue, by distributing the resources to watch over all running replicas. This however forced me to handle the case when one of my replicas fails and the replacement has to figure out which resources the dead replica was responsible for. I did this by labeling the resources with the pod-name of the responsible watcher.
