We have a Django application that should send messages and upload files from a web server to another server through an XML API. We need to do X-asynchronous file uploads, and then execute another XML API request when they finish uploading. I would also like the files to flow from the disk without loading them completely into memory. Finally, I needed to send the files as an application / octet stream to the POST body (and not the more typical MIME type data type), and I could not find a way to do this using urllib2 or httplib.
I ended up incorporating Twisted into the app. It seemed perfect for this task, and I'm sure I was able to write a beautifully clean implementation with a delay for each download. I use my own IBaseProducer to read data from a file in chunks and send it to the server in the body of the POST request. Unfortunately, I found out that the Twister reactor cannot be restarted, so I can’t just start it and then stop it whenever I want to upload files. Since Twisted seems to be more used for full-blown servers, now I'm wondering if that was right.
I'm not sure if I should: a) Configure the WSGI container (I am currently testing with manage.py) to start the Twisted thread at startup and use blockingCallFromThread to start the file download. b) Use Twisted as the WSGI container for the Django application. I assume that we will want to deploy Apache later, and I'm not sure what this means if we take this route. c) You can just Twisted and use a different approach to upload files. Kind of shame, because the Twisted deferred approach is elegant and works.
Which one should we choose or is there some other alternative?
source share