I have an application (running on an EC2 instance) that parses new log files coming into the S3 bucket. I have S3 notification settings in this bucket to put the message in the SQS queue in the PUT request. The goal is to transfer the recently received log file from S3 to an EC2 instance.
I think Apache Camel is the perfect candidate for this problem. I have the following questions / comments on the solution, and I would really appreciate some feedback on this:
We can have a Camel route to read messages from SQS, and then extract the object key from the JSON message. Is it possible to use this object key on another Camel route (using the S3 component) that copies this object from S3 to the local FS? In appearance, it seems that the Camel S3 component is able to process only new objects in the bucket, since only the bucket name is required in the URI. Thus, it may not be possible to copy a specific key from bucket S3. Also, I'm not sure if it is possible to have dynamic URIs in Camel, since the name of the object will be different every time.
If the Camel S3 component is really capable of handling new objects coming into the S3 bucket, then we do not need S3 notifications at all. But this problem (with its statelessness) has the problem that if for some reason this application goes down, it will skip the objects that were received during the downtime.
Another possibility is to use Camel to read messages from SQS, process these messages, and copy objects from S3 using the regular Java class. But this view does not fully utilize the strengths of Apache Camel.
source
share