AWS Lambda bound to S3 ObjectCreated event returns "NoSuchKey: the specified key does not exist:

Question

I upload a file from an Android device to an S3 bucket through this code

TransferUtility trasnferManager = new TransferUtility(s3, context); trasnferManager.upload(..,..,..); 

After that, I have a lambda trigger associated with the S3: ObjectCreated event.

When lambda is running, I try to get the file through the S3.getObject () function. Unfortunately, sometimes I get " NoSuchKey: the specified key does not exist: ". After that, lambda repeats a couple of times and successfully receives the file and continues to execute it.

In my opinion, is the lambda function executed before the file is available in S3? But this should not happen by design. The trigger should start after the file upload to S3 is complete.

According to the announcement on August 4, 2015 :

Amazon S3 buckets in all regions provide read-after-write sequencing for EMPTY new objects and possible consistency for overwriting EMPs and DELETES.

Consolidation after reading allows you to retrieve objects immediately after creation in Amazon S3.

But before that:

All regions except the US Standard (renamed US East (N. Virginia) ) support read-after-write consistency for new objects uploaded to Amazon S3.

My bucket is in the U.S. region (N. Virginia) , and it is created until August 4, 2015 . I do not know that this could be a problem ...

EDIT: 10/20/2016

According to documentaion , an EVENTUALLY CONSISTENT READ operation can return NO RESULTS even if two or more WRITE operations were completed before.

In this example, both W1 (record 1) and W2 (record 2) end before the start of R1 (read 1) and R2 (see 2). For sequential reading, R1 and R2 return color = ruby. Ultimately, read, R1 and R2 can return color = red, color = ruby ​​or without results, depending on the amount of time that has elapsed.

Answer 1

Sometimes, when files are large, they are downloaded using multi-component download and send a trigger for lambda before the file is fully loaded. Presumably this is due to an event that calls the lambda function. In the event field of the lambda function, make sure that you add both hyphenation and full multi-part loading to the event.

Answer 2

You can use the waiter S3 SDK to protect against this problem. After receiving the notification, we can guarantee that the object is actually there. For example, for the AWS JavaScript SDK, you can use the following snippet:

 s3.waitFor("objectExists", { Bucket: "<bucket-name>", Key: "<object-key>" }, callback); 

Please note that waitFor will increase the runtime of your Lambda, i.e. you will need to increase the timeout. According to the documentation, the check will be performed every 5 seconds up to 20 times. Therefore, setting a timeout to something around 1 minute should help avoid runtime exception exceptions.

Documentation Link: AWS JavaScript SDK S3 Class