2

I received a timeout when trying to perform 7000 * 3 inserts into MariaDB in 30 seconds with AWS Python Lambda. I have three ideas how to get around the 30 second API Gateway timeout:

  1. Convert the data to a file which MariaDB can import using LOAD_DATA_INFILE and let the Lambda do that.

  2. Write the data in a file to S3 and let another function write the data from S3 to MariaDB which then will not have the API Gateway timeout limit.

  3. Let the web client convert the file in JavaScript.

I am leaning towards the first strategy, to try and use LOAD_DATA_INFILE from the Lambda function. Do you agree?

2

A variation of method 2 is infinitely scalable:

  • write the data to s3.
  • from s3 trigger lambda to break s3 file into sqs queue events of good size
  • from sqs queue trigger lambdas to insert
  • sqs can be fifo queue if it matters.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.