Here is my situation:
I have a script that receives raw data, stores it in the database, gets the identifier of the element, and then sends it to the SQS azatone queue.
I have another script (written in PHP) that extracts raw data from db and processes it. (usually takes a few minutes.
The invalid part is retrieving messages from SQS for processing. The frequency of new processing data varies; it can go from a few items per hour to dozens of items per minute.
One way would be to have a cron job that requests SQS for new messages and starts the script processing (it must be multithreaded and written by something other than php. Maybe Python or Ruby). The problem with cron is that it is rather inefficient, in phases of high load it will be overloaded and in quiet phases it will make useless calls.
What I would like to have is some kind of multi-threaded listener that will receive messages and process them. One of the important requirements is that the raw data be processed in a timely manner, a couple of minutes after receiving no more.
Any thoughts on a better solution? Does SQS fit the task? What is the most efficient way to listen and process a queue?
multithreading php amazon-web-services amazon-sqs message-queue
applechief
source share