Geek Answers Handbook

SQS / SNS and architecture for one-time computing (instances of SP2 EC2)

I have an application that reads a message from SQS (let it call the p queue), performs expensive image processing (step # 1), downloads the result to S3 and removes the message from the p queue and then sends a notification to the SNS topic (this the SNS question directs the message to another queue called "q"). There is another application that reads “q” from the queue and performs the second stage of image processing (loads the result of step No. 1 from S3 and performs additional mathematical operations on this result).

I have a combination of regular instances + stain instances working with application # 1. I know that (due to the concept of SQS visibility timeout), if spot instances are closed during the image processing phase, SQS makes messages visible again to others consumers, so instances of EC2 without a spot will eventually do the job; the instances could not be completed due to system shutdown.

Now my question is: what happens if the spot instances are closed exactly after deletion, but before sending the SNS message? How can we recover from such an event?

# PSEUDO CODE
msg = read message from queue 
result = doWork(msg)
upload result to S3
delete msg 
publish to sns about result

Greetings!

+4
source share
1 answer

, A SQS , , SNS, . - , , , " ". SNS , .

-, , ( ), Idempotence: http://en.wikipedia.org/wiki/Idempotence

( ) , , , ,

: http://aws.amazon.com/sqs/faqs/#How_many_times_will_I_receive_each_message

Amazon SQS , " " . , , .

, , , , SQS , , SQS , SNS, SQS.

, , , , " , ", , , , . , .

+7

More articles:


All Articles