Is it better to perform delayed jobs using queues or at app level?

Question

I'm trying to scale up my push notification infrastructure. We currently use laravel app running on 2 ec2 instances horizontally scaled behind a load balancer, and each instance has a beanstalk queue engine that processes jobs created by laravel (including push notification jobs).

the problem is that this solution is not performant. Sometimes we send a marketing push to all our clients.. numbering in the thousands, and the time range difference between the first push notification and the last can be many hours.

I was considering what infrastructure would be good options. I considered AWS SQS, but it turns out that it's max delay job capacity is 15 minutes. In my business, customers can make pre-orders 20 hours before. So what we currently do is scheduled a delayed job 20 hours from now.

But then some people suggested that the delayed part should only happen at the application level (ie delay(work, timedelta(hours=20))), and that would allow me to use SQS anyways. Their reasoning is that delaying jobs is a business logic thing, and it's best handled at an application level and not an infrastructure level.

I'm not sure I agree.. and that's why I'm picking AWS elastic cache with redis. What do you guys think?

Answer 1

I would recommend using a scheduler to schedule job execution, rather than building logic to delay execution. Schedulers give you the added advantage of status updates. Consider using Simple Workflow Service from AWS.

Instead of using beanstalk queue engine, could you use Lambda to process your marketing jobs? Simple Workflow Service works with Lambda. You don't have to worry about scaling with Lambda.

Answer 2

This sounds on the fence with opinion based but I think there's an important thing that narrow enough the problem to be answered.

It is still an opinion but, as far as you're horizontally scaling, instances may appear and disappear at will, keeping the delay in memory is thus not an option, so at this point you'll have to workaround this and bake something like a notifications queue/table to keep what is to be sent at which time and periodically check what is to be sent.

Once that is said, you'll need a central place somewhere to keep the needed notifications out of the app instance themselves and as such you'll need a place to store them, it could be a redis service, a database table or whatever, you'll be coding it anyway.

When working on AWS, the services provided are already redundant and on high availability without much effort needed on your part, as such I'd go with an application code storing the notification somewhere with the time to send it and a lambda checking periodically which notification has to be sent, the notification can then be sent through SQS to take advantage of the retries in case of delivery failure or you'll have to handle it in the notification firing system.

The store part being elastic cache or something else is a matter of taste and team knowledge in my opinion, take whatever you're more comfortable with.
For the part firing the notifications I'd use something separated from the app itself, another app or a lambda checking periodically the 'cache' (DB or redis or whatever) for notifications to be sent.

This allows the app to be sure the notification once declared and acknowledged by the cache is properly registered and will be fired when needed and retried in case of problem. A last case is the impossibility to deliver the notification and you may have to keep track of this in the application to avoid sending too much notifications ending as dead letter but that's sliding out of scope for this question.