As part of my architecture, I want to create a deployment/service on my kubernetes cluster which reads message queues and open dockerized subscribers and then subsequently relay message to the original subscriber (if its an RPC pattern).
I was wondering what would be the best workflow to ensure minimal latency (esp since some workflows are RPC).
- I can create kubernetes JOBs but that would mean i read logs for the output of pods which would be relayed and then subsequently delete the job . But I worry this might be an overkill
- I can volume share the docker socket and my service can then run my images using docker client. But I am not if thats the right way to do so given that i would like to have proper tracking/versioning of my docker images esp since nodes on clusters are disposable
- Are there existing frameworks to handle my situation.