Flasks long procedures

Question

I need to do a long job in my Flask application. And I want to do this asynchronously. Just get started and then check the status from javascript.

I am trying to do something like:

@app.route('/sync') def sync(): p = Process(target=routine, args=('abc',)) p.start() return "Working..." 

But this creates non-existent gunsmith workers.

How can this be solved? Should I use something like celery?

Answer 1

There are many options. You can develop your own solution, use Celery or Twisted (I'm sure there are more ready-made options, but they are the most common).

Developing your internal solution is easy. You can use the multiprocessing module of the Python standard library:

• When a task arrives, you insert a row into your database with the identifier and status of the task.
• Then run the process to do the job, which updates the state of the row at the end.
• You may have a view to check if a task that actually checks the status in the appropriate one has completed.

Of course, you need to think about where you want to save the calculation result and what happens with errors.

Celery is also easy. It will look as follows. To define a function that runs asynchronously:

 @celery.task def mytask(data): ... do a lot of work ... 

Then, instead of directly invoking a task, such as mytask(data) , which will execute it immediately, use the delay method:

 result = mytask.delay(mydata) 

Finally, you can check if the result is available or not with ready :

 result.ready() 

However, remember that to use Celery you need to start an external workflow.

I never looked at Twisted, so I can’t tell you if this is more or less complicated than that (but it should be nice to do what you want too).

In any case, any of these solutions should work just fine with Flask. To check the result, it doesn't matter at all if you use Javascript. Just make a view that checks for JSON return status (you can use Flask jsonify ).

Answer 2

I would use a message broker like rabbitmq or activemq. The flask process adds jobs to the message queue, and a lengthy workflow (or pool or workflows) will execute jobs from the queue to complete them. The workflow can update the database so that the flash server can find out the current status of the job and pass this information to clients.

Using celery seems like a good way to do this.