Asynchronous WSGI with Twisted

Question

I am creating a web interface for a twisted application and would like to use WSGI rather than twisted.web directly (since the rest of the site is WSGI, and I already have a substantial WSGI base).

The Twisted documentation page I found about WSGIResource (http://twistedmatrix.com/documents/current/web/howto/web-in-60/wsgi.html) says: Like any other WSGI container, you you cannot do asynchronous actions in your WSGI applications, although this is a Twisted WSGI container.

Should this be true? Is there any less hacky way to do asynchronous request processing in twisted.web in WSGI - perhaps as part of another free software project? Suppose not, my plan is for WSGI threads to do their asynchronous work in the reactor thread and block by polling until data is available. This is not very.

If there is a fairly simple way to process WSGI requests asynchronously in twisted, I would like to hear it.

Answer 1

Why do you want to use WSGI and do asynchronous things? The advantage of WSGI is that you can deploy your application in any WSGI container. If you are starting to use the Twisted API to perform asynchronous actions, you can only deploy the application in the Twisted WSGI container.

You probably just need to use Twisted Web without WSGI for your asynchronous code.

Answer 2

Basically, WSGI is not internally incompatible with asynchronous program design; in fact, PEP 333 goes a considerable length to indicate how servers, applications, and middleware should work to support these kinds of things.

At the heart of this is the return of the iterator to the container. Each time asynchronous wsgi app_iter is called, it checks all its pending asynchronous tasks (database connections, etc.), and if any of them has data, app_iter gives some data; otherwise, it produces an empty string. To support this, the wsgi container will have to keep track of all requests in flight, and iterate through each one of them in order to get more data, in addition to servicing any other deferred work for which it is responsible.

Basically, very few wsgi applications or frameworks actually do this. almost always, wsgi frameworks are blocked for various reasons; reading files from disk or loading data from the database for any reason at all (most ORMs make this problem difficult.) The wsgi twisted container works under the assumption that since some wsgi applications are blocked, it is possible that any wsgi application can be blocked and therefore always runs them in a stream.

There are two things you can do; either explore your own twisted web structure, which is strong enough; or consider creating a wsgi wrapper to twist outside your own twisted container. Making sure that the wsgi application is actually asynchronous is certainly a prerequisite of the latter, but wsgi itself is pretty simple, a thin shell on top of http, and therefore it should be fairly light.