Why urllib.urlopen (url) fails while urllib2.urlopen (url) is running. What exactly is related to the server response?

Question

I just want to better understand what is happening here, I can of course "work around" the problem using urllib2.

import urllib import urllib2 url = "http://www.crutchfield.com/S-pqvJFyfA8KG/p_15410415/Dynamat-10415-Xtreme-Speaker-Kit.html" # urllib2 works fine (foo.headers / foo.read() also behave) foo = urllib2.urlopen(url) # urllib throws errors though, what specifically is causing this? bar = urllib.urlopen(url) 

http://pae.st/AxDW/ shows this code in action with the exception of / stacktrace. foo.headers and foo.read() work fine

[e-mail] ~ $ : curl -I "http://www.crutchfield.com/S-pqvJFyfA8KG/p_15410415/Dynamat-10415-Xtreme-Speaker-Kit.html"

 HTTP/1.1 302 Object Moved Cache-Control: private Transfer-Encoding: chunked Content-Type: text/html; charset=utf-8 Location: /S-FSTWJcduy5w/p_15410415/Dynamat-10415-Xtreme-Speaker-Kit.html Server: Microsoft-IIS/7.5 Set-Cookie: SESSIONID=FSTWJcduy5w; domain=.crutchfield.com; expires=Fri, 22-Feb-2013 22:06:43 GMT; path=/ Set-Cookie: SYSTEMID=0; domain=.crutchfield.com; expires=Fri, 22-Feb-2013 22:06:43 GMT; path=/ Set-Cookie: SESSIONDATE=02/23/2012 17:07:00; domain=.crutchfield.com; expires=Fri, 22-Feb-2013 22:06:43 GMT; path=/ X-AspNet-Version: 4.0.30319 HostName: cws105 Date: Thu, 23 Feb 2012 22:06:43 GMT 

Thanks.

Answer 1

This server is not deterministic and sensitive to the HTTP version. urllib2 - HTTP / 1.1, urllib - HTTP / 1.0. You can reproduce this by running curl --http1.0 -I "http://www.crutchfield.com/S-pqvJFyfA8KG/p_15410415/Dynamat-10415-Xtreme-Speaker-Kit.html" several times in a row. Sometimes you should see curl: (52) Empty reply from server output curl: (52) Empty reply from server ; that the urllib error is being urllib . (If you resubmit the request a bunch of times with urllib, this should be successful sometimes.)

Answer 2

I solved the problem. I just use urrlib now instead of urllib2 and everything works fine, thank you all :)