1

I am trying to use Redshift to UNLOAD data to S3 and then pick that data up with a Lambda function and process it. All of the constituent parts (the UNLOAD, the Lambda function) now work, however the Lambda event trigger (S3:ObjectPUT) is never firing. If I upload manually via the console, it all works. If I run the UNLOAD the data appears in the bucket, but the trigger event doesn't run.

I have tried creating / recreating the function (as per this forum thread - https://forums.aws.amazon.com/thread.jspa?messageID=768407&tstart=0), but to no avail.

Has anyone else had this - is there something I'm missing?

0

You did not tell how big data amounts you tested with, but if you test ok with little data block successfully and larger real situation fails, there may occur same as occurred here:

The larger files are almost certainly being uploaded via S3 Multipart Upload instead of a regular Put operation.

..

It sounds like the function is only subscribed to s3:ObjectCreated:Put events currently, and you need to add s3:ObjectCreated:CompleteMultipartUpload to the configuration.

  • Thanks. For future reference the size doesn't appear to matter. I created an inline function that just dumped the incoming event data, and even with a select 'foo' you still get the multipart upload. I think with bigger datasets that are split up you need to wait for the manifest, which does use a put. – Hugo Rodger-Brown Oct 8 '17 at 18:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.