How to save state for a long process called from Django?

Question

I am working on a Django application that allows the user to upload files. I need to do some server side processing on these files before sending them to Amazon S3 . After reading the answers to this question and this blog post, I decided that the best way to handle this would be to have my view handler call the Pyro method on the remote object to execute the processing asynchronously, and then immediately return the Http 200 for the client. I have this prototype and it seems to work well, however, I would also like to save the processing state so that the client can poll the application to see if the file has been processed and uploaded to S3.

I can handle the survey fairly easily, but I'm not sure where is the right place to store the state of the process. It should be writable by the Pyro process and available for viewing in my survey.

• I hesitate to add columns to the database for data that should really only persist for 30-60 seconds.
• I have considered using Django's low-level cache API and using the file identifier as the key, however, I don’t think this is actually what the cache infrastructure is intended for, and I’m not sure what unforeseen problems might arise when going along this route.
• Finally, I looked at the storage state in the Pyro object that is executing the processing, but then it still seems that I will need to add the logical column of the processing_complete database so that the view finds out whether to request the state from the Pyro object.

Of course, there are also some problems with data integrity in the state of decoupling from the database (what happens if the server goes down and all this data is in memory?). I should hear how more experienced web application developers would handle this processing in a stateful manner.

Answer 1

We do this by having the "Request" table in the database.

File .

.

200 " " - ​​ .

Django ORM. , Request. ( ) . , , , .

---

.

WSGI, . - REST POST ; .

REST. , . , , . .

, crontab, , . , 30 " ?" . script ( Apache mod_wsgi), "" script, WSGI, POST URL-, ( ).

, , POST. , , Request - , , , - .

Answer 2

, , - , .

, , , .

AMQP . Celery Carrot , RabbitMQ ZeroMQ.

, , .

Celery RabbitMQ . RabbitMQ , Celerye , .

octopy.

Answer 3

, , . , , . , , , . , - .

If you need something more powerful or more complex, I would look at something like Gearman .