AWS S3: launch multiple targets through S3 Notification when a file is received

Question

I want to configure the following event / notification configuration in the AWS S3 bucket: After receiving the file (s3: ObjectCreated: *), two targets are launched:

• SQS: Queue to file for detailed post-processing for a storage period of several days
• Lambda: for quick processing of fast metrics

When I try to configure the configuration using the AWS console, I get the following error message:

Configurations overlap. Configurations on the same bucket cannot share a common event type. : s3:ObjectCreated:*, s3:ObjectCreated:* 

I tried to configure the configuration through the AWS SDK (Java), as suggested by the user guide, but with a similar result:

 Caught an AmazonServiceException, which means your request made it to Amazon S3, but was rejected with an error response for some reason. Error Message: Configurations overlap. Configurations on the same bucket cannot share a common event type. (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: A0E8738522EA218F) HTTP Status Code: 400 AWS Error Code: InvalidArgument Error Type: Client Request ID: A0E8738522EA218F Error XML<?xml version="1.0" encoding="UTF-8"?> <Error><Code>InvalidArgument</Code><Message>Configurations overlap. Configurations on the same bucket cannot share a common event type.</Message><ArgumentName>Event</ArgumentName><ArgumentValue>s3:ObjectCreated:*, s3:ObjectCreated:*</ArgumentValue><RequestId>A0E8738522EA218F</RequestId><HostId>p4qYoIXi38u3Jl3p0xpI7TFWgs0ZxsqK89oDTTy8D/tbw39NnaIT99jIvHIxt4XliRFxqNWl32M=</HostId></Error> 

Answer 1

I suggest you publish S3 Notifications to the SNS Topic and sign the Lambda function and SQS Queue for this SNS Topic .

This architecture should help you achieve what you are looking for.

Answer 2

The best solution is likely to be to trigger SNS notification when files are uploaded to S3, and then use the SNS “fork” capabilities to send mutliple, simultaneous SQS messages, which can then be received and act independently.

Alternatively, if you want to process "step 2" only when "step 1" is processed, you can initiate one SQS message for "step 1", and then only upon successful completion of "step" 1, the final act of "step 1 "represents the dispatch of the second sqs event for" step 2 "for further processing.