Disclaimer: this is certainly not the best solution for backend workers, I'm just trying to be creative within the given constraints.
First, you'll need to be familiar with SQS's visibility timeout (documentation) for this to work.
Let's assume the following, then I'll walk through how it might work:
- We only have 1 request (henceforth "message") coming in at a time (10 per day, hopefully that's feasible)
- We can take up to 12 minutes to process a message in the worst-case scenario
Next, some configuration changes:
- We configure the ASG to scale desired count to 0 when
ApproximateNumberOfMessagesVisible = 0 for 5 minutes.
- We configure the ASG to scale desired count to 1 when
ApproximateNumberOfMessagesVisible >= 1.
- We set all messages
VisibilityTimeout to 10 minutes.
A failure scenario might work out like this:
- At 0 minutes, 1 message is being processed by the worker. There are now 0 other messages in the queue,
ApproximateNumberOfMessagesVisible = 0
- At 2 minutes we are now assuming that the worker is now stuck processing its in-flight message. No action is taken yet.
- At 5 minutes our ASG scale-down event triggers, and desired capacity is set to 0 (because there is no pending work)
- At 10 minutes, SQS automatically re-enqueues the message due to the visibility timeout.
ApproximateNumberOfMessagesVisible = 1
- Shortly after, our ASG scale-up event triggers, and desired capacity is set to 1 (because there is pending work)
The 3-minute and 5-minute gaps are somewhat arbitrary, they merely highlight the need for a brief waiting period for such a fickle system.
The best worker is one that exits its process instead of getting stuck. That would allow for process monitoring on EC2, or for containerization. Were your worker an ECS service, it could scale up when there is work, recover from failures, and scale down afterwards. Refactoring your worker might be less effort than ongoing manual intervention for my hacky solution, and save you money!