Setting referral url in python urllib.urlretrieve

Question

I am using urllib.urlretrieve in Python to load websites. Although some websites do not seem to want me to download them unless they have a proper referrer from their website. Does anyone know how I can install the referrer in one of the Python libraries or external.

Answer 1

 import urllib2 req = urllib2.Request('http://www.example.com/') req.add_header('Referer', 'http://www.python.org/') r = urllib2.urlopen(req) 

adopted by http://docs.python.org/library/urllib2.html

Answer 2

urllib makes it difficult to send arbitrary headers with a request; you can use urllib2 , which allows you to create and send a Request object with arbitrary headers (including, of course, sadly written ;-) - Referer ). It does not offer urlretrieve , but you just urlopen , like you, and copy the resulting file-like object to disk if you want (directly or, for example, via shutil ).

Answer 3

Alternatively, using urllib2 with build_opener , you can do this:

 import urllib2 opener = urllib2.build_opener() opener.addheaders = [('Referer', 'http://www.python.org/')] opener.open('http://www.example.com/')