2

I have a processing server that processes requests in an SQS queue, only about 10 per day, each taking around 2 minutes to process.

Sometimes the process gets "stuck" and it would be ideal to terminate the single server in the autoscaling group, but keep the autoscaling size to 1, as the problem is often resolved with a fresh server created by autoscaling.

Can this be done with autoscaling configuration alone, or do I need to create a separate process with step or lambda functions?

Improve this question

2 Answers 2

Reset to default
4
+50

Disclaimer: this is certainly not the best solution for backend workers, I'm just trying to be creative within the given constraints.

First, you'll need to be familiar with SQS's visibility timeout (documentation) for this to work.

Let's assume the following, then I'll walk through how it might work:

  • We only have 1 request (henceforth "message") coming in at a time (10 per day, hopefully that's feasible)
  • We can take up to 12 minutes to process a message in the worst-case scenario

Next, some configuration changes:

  • We configure the ASG to scale desired count to 0 when ApproximateNumberOfMessagesVisible = 0 for 5 minutes.
  • We configure the ASG to scale desired count to 1 when ApproximateNumberOfMessagesVisible >= 1.
  • We set all messages VisibilityTimeout to 10 minutes.

A failure scenario might work out like this:

  1. At 0 minutes, 1 message is being processed by the worker. There are now 0 other messages in the queue, ApproximateNumberOfMessagesVisible = 0
  2. At 2 minutes we are now assuming that the worker is now stuck processing its in-flight message. No action is taken yet.
  3. At 5 minutes our ASG scale-down event triggers, and desired capacity is set to 0 (because there is no pending work)
  4. At 10 minutes, SQS automatically re-enqueues the message due to the visibility timeout. ApproximateNumberOfMessagesVisible = 1
  5. Shortly after, our ASG scale-up event triggers, and desired capacity is set to 1 (because there is pending work)

The 3-minute and 5-minute gaps are somewhat arbitrary, they merely highlight the need for a brief waiting period for such a fickle system.

Reference:

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html

HOWEVER...

The best worker is one that exits its process instead of getting stuck. That would allow for process monitoring on EC2, or for containerization. Were your worker an ECS service, it could scale up when there is work, recover from failures, and scale down afterwards. Refactoring your worker might be less effort than ongoing manual intervention for my hacky solution, and save you money!

Improve this answer
6
  • Thank you! I need to think about this a little bit. The main problem is throughput. The 10 messages per day usually come in in 1 batch or 2 batches, so this will cause a lot of throughput delay.
    – jdog
    Commented Jun 10, 2020 at 22:41
  • Hmm, I suppose my hacky solution would work if instead of scaling to 1 instance you scaled to N, where N is the most batches you would see at one time. It's likely to be wasteful (in the sense that it over-provisions at times) but at least it's still cheaper than running 1 full-time like you are now.
    – Woodland
    Commented Jun 12, 2020 at 17:44
  • The answer clarified/ reminded me that I can simply terminate my instance when messages have been in the queue for too long, because I can expect all messages to be processed within a certain timeframe based on my assumption of large gaps in processing traffic. Even with consistent processing traffic, my processing server would simply be terminated and rebuild every x hours.
    – jdog
    Commented Jun 15, 2020 at 21:07
  • 1
    there is another piece of technical debt in this system, which is that the processing is simply running a macro in Excel triggered by Windows task scheduler. Because of its minimum 5 minute time resolution, processing doesn't start on average for 2.5 minutes after boot up is complete. Windows instances being billed at 1 hour minimum also complicate the trade off.
    – jdog
    Commented Jun 15, 2020 at 21:09
  • Good point, especially the Windows billing consideration. Sounds like quite the difficult system, I don't envy you!
    – Woodland
    Commented Jun 16, 2020 at 17:47
1

Use AWS Automation document on state change to modify autoscaling group to 0 and return to 1 after. It should terminate and recreate.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.