Strategies to work around 30 sec timeout

Question

I received a timeout when trying to perform 7000 * 3 inserts into MariaDB in 30 seconds with AWS Python Lambda. I have three ideas how to get around the 30 second API Gateway timeout:

1. Convert the data to a file which MariaDB can import using LOAD_DATA_INFILE and let the Lambda do that.

2. Write the data in a file to S3 and let another function write the data from S3 to MariaDB which then will not have the API Gateway timeout limit.

3. Let the web client convert the file in JavaScript.

I am leaning towards the first strategy, to try and use LOAD_DATA_INFILE from the Lambda function. Do you agree?

Answer 1

A variation of method 2 is infinitely scalable:

• write the data to s3.
• from s3 trigger lambda to break s3 file into sqs queue events of good size
• from sqs queue trigger lambdas to insert
• sqs can be fifo queue if it matters.