Urllib2 does not use proxy (Fiddler2) installed using ProxyHandler

Question

I have Fiddler2 listening on 0.0.0.0:8888.

try: data = '' proxy = urllib2.ProxyHandler({'http': '127.0.0.1:8888'}) //also tried {'http': 'http://127.0.0.1:8888/'} opener = urllib2.build_opener(proxy) urllib2.install_opener(opener) req = urllib2.Request('http://www.google.com') response = urllib2.urlopen(req) the_page = response.read() print the_page except Exception, detail: print "Err ", detail 

I do not see GET or any google request in Fiddler (but I see other requests) is there a way to debug it? it looks like python bypasses Fiddler or ignores proxies.

I also configured WinHTTP to work with Fiddler -

 C:\Windows\system32>netsh winhttp set proxy 127.0.0.1:8888 Current WinHTTP proxy settings: Proxy Server(s) : 127.0.0.1:8888 Bypass List : (none) 

Is this really a question if requesting an SSL address? (Fiddler supports https)

Thanks!

Answer 1

Perhaps you can work with the opener directly, rather than installing it. turn on the Fiddler proxy listener on 8008 (I use WebScarab, but they are probably the same), then try this code for sure (there are also cookies that you don’t need, but let's try it as it is and narrow it down later)):

 cj = cookielib.MozillaCookieJar(cookie_filename) if os.access(cookie_filename, os.F_OK): cj.load() proxy_handler = urllib2.ProxyHandler({'https': 'localhost:8008'}) opener = urllib2.build_opener( proxy_handler, urllib2.HTTPCookieProcessor(cj) ) opener.addheaders = [ ('User-agent', ('Mozilla/4.0 (compatible; MSIE 6.0; ' 'Windows NT 5.2; .NET CLR 1.1.4322)')) ] auth = urllib.urlencode({'email':email,'pass':passw}) data = opener.open('https://login.facebook.com/login.php',data=auth) 

so what I am doing is different: direct use of the opening device, changing the port to 8008, adding cookies and using WebScarab. let me know which one did the trick for you ...

Answer 2

proxy_bypass_registry in urllib.py does not handle the value of the ProxyOverride registry correctly: it treats an empty override as *, that is, bypasses the proxy for all hosts. This behavior is not consistent with other programs (e.g. Chrome).

There are a number of possible workarounds:

• Set urllib.proxy_bypass = lambda h: 0 to disable bypass checking.
• Specify the proxy server settings in the http_proxy environment http_proxy ( proxy_bypass_registry is not called in this case).
• In Fiddler2, go to Tools-> Fiddler Options ...-> Connections, remove the end semicolon from the value in the "IE should bypass Fiddler for ..." field and restart Fiddler2.

Answer 3

In Fiddler2, go to Tools → Fiddler Options ... → Connections, remove the end semicolon from the value in IE, to bypass Fiddler for ... and restart Fiddler2.

This solution certainly works for me when I use the urllib2 proxy, however I still don't understand why its removal can be solved with a semicolon.

Answer 4

btw, you need to use http://www.google.com/ instead of http://www.google.com so that the violinist can understand that you are requesting 'get /'

otherwise the violinist cannot understand uri. (u may be refused upon receipt of 504).