Geek Answers Handbook

Python CGI Queue

I am working on a fairly simple CGI with Python. I am going to put it in Django etc. The general setup is fairly standard server (i.e., the calculation is performed on the server):

  • The user downloads the data files and clicks the "Run" button
  • The server automatically starts tasks behind the scenes, using a lot of RAM and processor power. ~ 5-10 minutes later (average use case), the program terminates by creating its output file and some .png shape files.
  • The server displays a web page with numbers and summary text.

I do not think that hundreds or thousands of people will use it right away; however, since the calculation takes place with sufficient RAM and processor power (each instance launches the most CPU-intensive task using the Python Pool ).

I was wondering if you know whether it is worth using a queuing system. I stumbled upon a Python module called beanstalkc , but on the page he said it was a "in-memory" system.

What does “in memory” mean in this context? I'm worried about memory, not just CPU time, so I want only one job to execute at a time (or it was stored in RAM, regardless of whether it gets CPU time).

In addition, I tried to decide

  • the results page (served by CGI) should tell you its position in the queue (until it starts, and then displays the actual results page)

    OR

  • the user must send their email address to CGI, which will email them a link to the results page when it is complete.

What, in your opinion, is the appropriate design methodology for CGI light traffic for this kind of problem? Advice is greatly appreciated.

+1
source share
1 answer

Definitely use celery . You can start the amqp server, or I think you can sue the database as a queue for messages. It allows you to run tasks in the background and can use several working machines to perform processing if you want. It can also do base-based cron jobs if you use django-celery

It is as simple as running a task in the background:

 @task def add(x, y): return x + y

In the project, I have a distribution kit for working on 4 machines, and it works fine .

+1
source

More articles:


All Articles