Challenges of using AWS Lambda chain

Question

Context: I have a pipeline of 6 lambda functions (chained together), triggered by an SNS notification which is generated whenever a file lands on S3. This pipeline essentially takes the file(few GBs), filters it (Spark cluster is created to run the job, then deleted at the end), and inserts it into a DB. Lambdas are orchestrating the flow.

Issues: If one Lambda fails, the chain breaks hence no effective failure handling. Secondly, we experience timeouts if a polling/computation takes longer than 5 minutes, so no effective retry. It takes a long time to test/debug an issue if a lambda fails. Also there is no visibility, say for example how many jobs failed and how many passed? we dont know. Getting a bunch of SNS notifications on email is not very effective/helpful. If the chain breaks, we cannot perform cleanup operations like deleting SPark cluster or housekeeping steps.

My Questions: Is AWS Step Functions a good choice for solving the above issues? When would you not use a Step Function service? If you cannot invoke Step Function through SNS, then what would be the best way to call it whenever a file lands on S3? Feel free to share any other approach to easily and effectively tackle this usecase.

Answer 1

I am using lambda in response to objects appearing on S3 buckets. For me in my particular use case, there are around 600 triggers a day. I have written our lambda functions to use their stdout as their log file, writing all key events to standard out in a semi-structured textual sentence mixed with key/value pairs for data values, and then feed the cloudwatch logs into splunk.

Here is what I have found...

1. Lambdas will (occasionally) be triggered multiple times for the same object.
2. Lambdas will (sometimes) take noticeably longer to run than normal.
3. To see / diagnose issues caused by concurrent invocations that interfere with each other, the AWS Web Console doesn't do it. You need a tool like splunk.

However, does the value of this work provide enough revenue to cover the costs of using ECS(Elastic Container Service) for your process management? If you are able to run an ECS container, you could trigger the lambda on S3 object creation to create a message on an SQS message queue. Your ECS container is listening to your queue and when a message arrives, processes it just like within your lambda. Its output to trigger the next stage is to pop a message onto the SQS queue handling the next task.

This will solve a few problems for you, such as escaping the 5 minute maximum for Lambdas, and creating an easy way to retry failures.

However, it will create a few new issues too.

• Creating and deploying docker images is more work than editing lambda on the AWS console.
• Infrastructure costs are more fixed in nature, rather than proportional to workload.

Answer 2

Yes, Step Functions seems like a good fit for your use case.

Step Functions standard workflows guarantees exactly-once execution and provides a step-by-step history of your workflows, hence making it easy to debug and giving you the visibility that you are looking for. You can additionally use retry/catch functionality and SNS service integration.

You can use EventBridge to trigger Step Functions from S3.

Resources: https://docs.aws.amazon.com/eventbridge/latest/userguide/eventbridge-tutorial-ecs.html (change target to Step Functions instead of ECS) https://docs.aws.amazon.com/step-functions/latest/dg/concepts-error-handling.html https://docs.aws.amazon.com/step-functions/latest/dg/connect-sns.html

I work for AWS but all opinions are my own.

Answer 3

I'm still getting into AWS Lambda so I'm not sure if it will help you but I started using a lambda performance monitoring tool for my functions and it seemed to be working well. I get a message if something breaks in 60 seconds (+ daily reports but that's less important) and it helped me figure out what broke.

Now, I don't know if you can indeed use Step Functions to fix your issues but I do know you can set up events that trigger a Lambda function whenever a file gets to S3. Here's a link to s3 docs that might help.