How to create a .NET / MVC 3 Job / Queue system?

Question

Problem

I once wrote an internal application to read data from an internal database, and then I take this data and send it to a web service. The application is very simple, using a single stream and synchronous HTTP requests.

The scope of this application has now changed, and it is trying to produce much more data than it ever intended for promotion. If he reads 1000 records from our internal database, he will transfer them all to a single POST HTTP mail, putting a lot of CPU load on the servers that host the web service that receives the data. Problems also arise when the web service encounters an error while processing one of the entries in the POST. The XML response does not indicate which particular record failed, so I am limited by the visibility of the success of my request.

How i would like to fix it

I am going to redesign my application to be more reliable and more attentive to the servers hosting the web service. In particular, I would like to have a worker who collects records from an internal database every 15 minutes and turns them into jobs. These jobs will be serialized and stored in a queue (possibly a database table). I would like my application process to process the queue using multiple worker threads (is this a good idea?). The thread should pull the job out of the queue and process it by making an asynchronous HTTP POST for the web service up and down. Depending on the status of the request, the task will result in SUCCESS, MALFUNCTION, TIME or OPEN. The task will be updated in the database, the process will be logged, and then the workflow will go to the next task and become inactive if the task queue is empty.

I am not an architect, so I do not know how best to implement something like this. Here are some specific questions I have in design.

• I read some negative things about multithreading in the .NET MVC environment. Should I avoid using multiple threads since I am not doing anything that is really processor intensive?
• Quartz.NET looks like it can do a lot of cool things. Should I look at using Quartz.NET for something like this?
• Is my project reasonable? If not, how can this be improved?
• How would you develop a system to achieve the goals of a new application?

I know this is a broad question, but I hope that I have clearly stated my goals. Thank you in advance.

Answer 1

Have you considered MSMQ for this? You click messages in the queue, read every N minutes, and turn on redundancy if power failures occur, etc. If you are in a load-balanced environment, you can send messages to the general queue.

In response to questions:

I read some negative things about multithreading in the .NET MVC environment. Should I avoid using multiple threads since I am not doing anything that is really processor intensive?
You are not recommended to use ThreadPools in ASP.NET, therefore the same applies to MVC. It can throttle your application.

Quartz.NET looks like it can do a lot of cool things. Should I look at using Quartz.NET for something like this?
This is a replacement for scheduling, not queuing, akin to cronjobs.

Is my design reasonable? If not, how can this be improved?
The serial part sounds great, the sound of SUCCESS, FAILURE, TIMEOUT or ABORTED sounds good. As already mentioned, MSMQ will save you the extra redundancy record and message queuing system.

How would you develop a system to achieve the goals of a new application?
A service that reads from the message queue so often and performs the action you want to do. You can also take a look at SQL Server Message Broker as an alternative to MSMQ. MSMQ does not have very good built-in tools to manage it, so you will need to build on top of it. However, it has the entire .NET assembly as part of the built-in structure for its use.

Since you are using .NET 4, you can also take advantage of parallel tasks instead of manually controlling the flows of the HTTP sending part of your system.