How to ensure equitable distribution of SQS messages in a distributed systems setting?

Question

I have multiple servers, each having a script polling an SQS queue [all polling the same queue].

So, is there any way I can ensure an equitable distribution of messages to all those clients [i.e. my worker servers here]. Like for example, if there are 100 messages in the queue, then 20-20-20-20-20 if there are 5 workers, and so on.

Can AWS ELB (Elastic Load Balancer) help me do it? If yes, then how? If not, then is there an alternate service in the AWS ecosystem which can help me do it?

Or am I overthinking this? I mean, can this be solved straightforwardly in the polling script? [Please keep in mind the race conditions involved due to multiple clients polling a single queue]

Answer 1

If there are 100 messages in the queue and 5 consumers, the initial distribution will be no more than 10-10-10-10-10.

A single response can never return more than 10 messages.

This seems like a non-issue.

Race conditions related to multiple consumers should be a non-issue as well. SQS is designed for multiple simultaneous consumers.

Use long polls and a 20 second max wait timer and be amazed. (No, a 20 second wait does not delay messages by 20 seconds. It doesn't delay them at all. You sort of need to see it in action to really understand how it works.)

You're definitely overthinking some things, I suspect.

Answer 2

Good architecture in how you use SQS queues will solve your problems. If we assume there are, say, 3 minutes of processing per message, then you can almost guarantee equal distribution of the messages as this is very large compared to the time needed to poll the queue, if you delete the message from the queue only after it has been processed.

Do be aware there is a 12 hour Visibility Timeout limit on any SQS message, so if you dont delete it by then, it will re-appear on the queue. I suspect this is probably not a limitation for you, but do keep it in mind.

Answer 3

Long polling is always beneficial since it results in higher performance at reduced cost for the majority of use cases. Unfortunately, you cannot control the number of messages each worker receives from the queue due to the distributed nature of the queue. But there are some client-side workarounds that can help you in balancing the load for workers.

So, this is what we did as a workaround for this:

As one of the workaround, the poller script can control the number of messages each worker receives. A threshold can be set for maximum number of messages each worker can handle. This threshold can be a dynamic value and would probably be ApproximateNumberOfMessagesVisible divided by number of pollers/poller scripts. You can then keep the visibility timeout to be any lower value so if all the poller scripts are long polling at the same time, one of the pollers grabs the message, decides it is overloaded based on threshold, does not delete the message, the message goes back to the queue and it could be grabbed by other pollers who still has the capacity to grab the message. The threshold parameter can be fine tuned to meet the application's needs.

---

Also, having failover mechanism would help too, like how the answers in this post describe. However, I can't afford to have failover queues in a distributed architecture, as it would increase the complexity. So, the above workaround was a better idea for my team.