Download multiple parts of the same file at the same time with Python?

Question

I know how to use urllib to upload a file. However, it is much faster if the server permits it, to simultaneously download several parts of the same file, and then combine them.

How do you do this in Python? If you cannot do this easily with the standard lib, any lib that would allow you to do this?

Answer 1

PycURL can do this. PycURL is the Python interface for libcurl. It can be used to retrieve objects identified by URLs from a Python program, similar to the python urllib module. PycURL is mature, very fast and supports many features.

Answer 2

While I agree with Gregory’s suggestion to use an existing library, it's worth noting that you can do this using the Range HTTP header. If the server accepts byte range requests, you can run multiple threads to load multiple parts of the file in parallel. This fragment, for example, will load only bytes 0..65535 of the specified file:

 import urllib2 url = 'http://example.com/test.zip' req = urllib2.Request(url, headers={'Range':'bytes=0-65535'}) data = urllib2.urlopen(req).read() 

You can determine the size of the remote resource and see if the server supports range requests by sending a HEAD request:

 import urllib2 class HeadRequest(urllib2.Request): def get_method(self): return "HEAD" url = 'http://sstatic.net/stackoverflow/img/sprites.png' req = HeadRequest(url) response = urllib2.urlopen(req) response.close() print respose.headers 

The above prints:

 Cache-Control: max-age=604800 Content-Length: 16542 Content-Type: image/png Last-Modified: Thu, 10 Mar 2011 06:13:43 GMT Accept-Ranges: bytes ETag: "c434b24eeadecb1:0" Date: Mon, 14 Mar 2011 16:08:02 GMT Connection: close 

This shows that the file size is 16542 bytes ( 'Content-Length' ), and the server supports byte ranges ( 'Accept-Ranges: bytes' ).