Running AWS lambda in serial (for a given parameter)

Question

I'm not sure if this is the best place to ask this. But we've currently got something in Microsoft Azure where we have their serverless functions (Azure Functions) running in serial based off a particular key, say a user ID.

So I want these functions to be called in parallel for different users, but in serial for each individual user. I spent ages trying to do this with locking behaviour in the function logic but it was always causing issues until I discovered with Azure, I simply make my Azure Function trigger from Service Bus, and use the userId as the Service Bus session id. Then Azure makes sure only one 'session' is running at once.

We're looking at moving stuff over to AWS and Lambda functions, but there's so many services - AWS Firehose, Simple Queue, CloudWatch Events, Step Functions, Kinesis Data Stream. In my initial searching I couldn't find any references to something similar (but everything is called different things, and I can spend hours reading about a service and find it's totally nothing to do with what I'm trying to accomplish)

Does anyone know if any of them offer that kind of "serial for a given property" kind of behaviour out of the box?

Answer 1

Use SQS with Lambda to accomplish this.

SQS supports 2 types of queues: Standard & FIFO. FIFO is appropriate for your use case. Create a FIFO queue & attach a Lambda function trigger to it. This causes the Lambda function to be triggered when a message arrives in the SQS queue. When you send a message to a FIFO queue, set the userId as the message group ID. This will ensure that messages from an individual user are only processed serially but messages from different users are processed in parallel.

Here's the relevant section from AWS Lambda documentation:

For FIFO queues, Lambda sends messages to your function in the order that it receives them. When you send a message to a FIFO queue, you specify a message group ID. Amazon SQS ensures that messages in the same group are delivered to Lambda in order. Lambda sorts the messages into groups and sends only one batch at a time for a group. If the function returns an error, all retries are attempted on the affected messages before Lambda receives additional messages from the same group. Your function can scale in concurrency to the number of active message groups.