I am developing my first Amazon AWS project, and I could use some help in handling the queue.
This service accepts processing jobs, either through the ASP.net web API service, or through the GUI website (which simply calls the API). Each task has one or more files associated with it and some rules about the type of task. I want to queue every job as it appears, presumably using AWS SQS. Then the tasks will be processed by the "worker", which is a python script with a .Net wrapper. The Python script is an existing batch processor that cannot be changed / configured for AWS, therefore a .Net wrapper that manages parts of AWS and passes the correct parameters to python.
The problem is that we will not have a huge number of tasks, but each work is somewhat intense. One of the reasons for switching to AWS was to minimize infrastructure costs. I plan that the frontend website (web API + ASP.net MVC4 website) runs on an elastic beanstalk. But I would prefer not to have a specialized work machine, which always conducts an online survey of jobs, because these workers should be a little “stronger” copy (for processing), and it will cost us a lot to basically do nothing.
Is there a way to run the web part on beanstalk and then start the workflow only if there are elements in the queue? I understand that I can have a microcontroller instance that is always an online survey, and then it controls the promotion of calculations, but it even seems that it is not necessary. Can I run EC2 instances based on non-zero SQS queue size? Thus, basically the web api adds work to the queue, something looks at the queue and sees non-zero, this starts working EC2, it spins up and examines the queue at startup. It is processed until the queue becomes empty, then something starts it at the end of the work.
source share