Geek Answers Handbook

Handling encoded HTTP encoding with django

I have a problem handling http chunked transfer encoding.

I use:

  • Apache. Plugin
  • mod_wsgi.
  • Django

django is capable of handling a reqular http request with a content length header field, but when it comes to processing TE (Transfer-Encoding), chunked or gzip, it returns an empty result.

I think of two approaches:

  • Making some changes to the python django.wsgi file
  • Add some middleware python file in django to intercept any chunked http request, convert it to requestelar http request with content length header field, then transfer it to django where it can handle it.

Anyone can help with any of the above 2 options (more options are of course welcome)

Thanks!

This is an extension of my question after Graham's first answer:

First of all, thanks for your quick reply. The customer that is used is Axis, which is part of another company system that communicates with ours. I had the WSGIChunkedRequest On set, I also made some changes to my wsgi wrapper as follows:

def application(environ, start_response): 

  if environ.get("mod_wsgi.input_chunked") == "1": stream = environ["wsgi.input"] print stream print 'type: ', type(stream) length = 0 for byte in stream: length+=1 #print length environ["CONTENT_LENGTH"] = len(stream.read(length)) django_application = get_wsgi_application() return django_application(environ, start_response)

but it gives me these errors (extracted from apache error.log file):

 [Sat Aug 25 17:26:07 2012] [error] <mod_wsgi.Input object at 0xb6c35390> [Sat Aug 25 17:26:07 2012] [error] type: <type 'mod_wsgi.Input'> [Sat Aug 25 17:26:08 2012] [error] [client xxxxxxxxxxxxx] mod_wsgi (pid=27210): Exception occurred processing WSGI script '/..../wsgi.py'. [Sat Aug 25 17:26:08 2012] [error] [client xxxxxxxxxxxxx] Traceback (most recent call last): [Sat Aug 25 17:26:08 2012] [error] [client xxxxxxxxxxxxx] File "/..../wsgi.py", line 57, in application [Sat Aug 25 17:26:08 2012] [error] [client xxxxxxxxxxxxx] for byte in stream: [Sat Aug 25 17:26:08 2012] [error] [client xxxxxxxxxxxxx] IOError: request data read error

What am I doing wrong?!

+7
source share
2 answers

This is not a Django issue. This is a limitation of the WSGI specification itself to the extent that the WSGI specification prohibits the use of content with the requested request, which requires the value CONTENT_LENGTH for the request.

When using mod_wsgi, there is a switch to enable non-standard support for the content with the requested request, but this means that your application is not compatible with WSGI, plus it will require a special web application or WSGI wrapper, since it is still not going to work with Django.

An option in mod_wsgi for the allowed content of the request:

 WSGIChunkedRequest On

Your WSGI shell must call wsgi.input.read () to get all the content, create an instance of StringIO with it and use it to replace wsgi.input, and then add a new CONTENT_LENGTH value to the environment with the actual length before calling the wrapped application.

Note that this is dangerous because you do not know how much data is being sent.

Which client do you use anyway that only supports batch request content?

UPDATE 1

Your code has been broken for many reasons. You should use something like:

 import StringIO django_application = get_wsgi_application() def application(environ, start_response): if environ.get("mod_wsgi.input_chunked") == "1": stream = environ["wsgi.input"] data = stream.read() environ["CONTENT_LENGTH"] = str(len(data)) environ["wsgi.input"] = StringIO.StringIO(data) return django_application(environ, start_response)

Please note that this will not help with the contents of the gzip'd request. To do this, you will need an additional check to see when the content encoding has been compressed, and then do the same as above. This is because when the data is not compressed by Apache, the length of the content changes, and you need to recount it.

+9
source

Now everything works smoothly, the problem was in daemon mode, since it does not work with chunked http traffic, maybe in mod_wsgi 4 - according to Graham Dumpleton. So, if you have this problem, switch mod_wsgi to native mode.

As a modification of Graham's code in a wsgi wrapper, there are 2 options where you can read a stream buffered in an environment variable:

First:

 try: while True: data+= stream.next() except: print 'Done with reading the stream ...'

Second:

 try: data+= stream.read() except: print 'Done with reading the stream ...'

the first stub of the code was able to read the buffer in daemon mode, but it stopped somewhere, and the program did not continue to work (which confused me a bit, as I expected it to work beautifully), and the other stub code crashed with IOError and worked only in native mode.

Another thing to add, updating from 3.3 to 3.4 did not solve the problem, so you need to enable swtich in native mode.

These are my results and observations. If you have comments, additions or corrections, please feel free to.

Thanks!

+3
source

More articles:


All Articles