Python Proxies Proxy

Question

Just a short, simple example of the excellent Requests for Python.

I cannot find in the documentation what the variable 'proxies' should contain. When I send him a dict with the standard value "IP: PORT", he rejects it by requesting 2 values. So, I think (because this is not covered in the docs) that the first value is ip and the second is the port?

The docs mention only the following:

proxies - (optional) The dictionary mapping protocol for the proxy URL.

So, I tried this ... what should I do?

proxy = { ip: port} 

and should I convert them to some type before putting them in a dict?

 r = requests.get(url,headers=headers,proxies=proxy) 

Answer 1

Syntax proxies ' = {"protocol":"ip:port", ...} . With it, you can specify different (or the same) proxies for requests using the http, https and ftp protocols:

 http_proxy = "http://10.10.1.10:3128" https_proxy = "https://10.10.1.11:1080" ftp_proxy = "ftp://10.10.1.10:3128" proxyDict = { "http" : http_proxy, "https" : https_proxy, "ftp" : ftp_proxy } r = requests.get(url, headers=headers, proxies=proxyDict) 

Derived from documentation requests :

Options:
method - the method for the new Request object.
url - URL for the new Request object.
...
proxies - (optional) The dictionary maps the protocol to the proxy server URL .
...

---

On linux, you can also do this using the environment variables HTTPS_PROXY , HTTPS_PROXY and FTP_PROXY :

 export HTTP_PROXY=10.10.1.10:3128 export HTTPS_PROXY=10.10.1.11:1080 export FTP_PROXY=10.10.1.10:3128 

On Windows:

 set http_proxy=10.10.1.10:3128 set https_proxy=10.10.1.11:1080 set ftp_proxy=10.10.1.10:3128 

---

Thank you, Jay for pointing this out:
Syntax changed using 2.0.0 queries .
You will need to add the schema to the URL: http://docs.python-requests.org/en/latest/user/advanced/#proxies

Answer 2

I found that the urlib has really good code to pick up the proxy server settings, and they turned out to be in the correct form for direct use. You can use this as:

 import urllib ... r = requests.get('http://example.org', proxies=urllib.getproxies()) 

It works very well and urllib knows about getting Mac OS X and Windows settings.

Answer 3

Here you can refer to the proxy documentation .

If you need to use a proxy server, you can configure individual requests with the proxies argument to any request method:

 import requests proxies = { "http": "10.10.1.10:3128", "https": "10.10.1.10:1080", } requests.get("http://example.org", proxies=proxies) 

To use HTTP Basic Auth with your proxy, use the syntax http: // user: [e-mail]/ :

 proxies = { "http": "http://user:pass@10.10.1.10:3128/" } 

Answer 4

The accepted answer was a good start for me, but I kept getting the following error:

 AssertionError: Not supported proxy scheme None 

The fix was to specify http: // in the proxy url:

 http_proxy = "http://194.62.145.248:8080" https_proxy = "https://194.62.145.248:8080" ftp_proxy = "10.10.1.10:3128" proxyDict = { "http" : http_proxy, "https" : https_proxy, "ftp" : ftp_proxy } 

I would be wondering why the original works for some people, but not for me.

Edit: I see that the main answer is now updated to reflect this :)

Answer 5

here is my base class in python for the request module with some proxy configurations and a stopwatch!

 import requests import time class BaseCheck(): def __init__(self, url): self.http_proxy = "http://user:pw@proxy:8080" self.https_proxy = "http://user:pw@proxy:8080" self.ftp_proxy = "http://user:pw@proxy:8080" self.proxyDict = { "http" : self.http_proxy, "https" : self.https_proxy, "ftp" : self.ftp_proxy } self.url = url def makearr(tsteps): global stemps global steps stemps = {} for step in tsteps: stemps[step] = { 'start': 0, 'end': 0 } steps = tsteps makearr(['init','check']) def starttime(typ = ""): for stemp in stemps: if typ == "": stemps[stemp]['start'] = time.time() else: stemps[stemp][typ] = time.time() starttime() def __str__(self): return str(self.url) def getrequests(self): g=requests.get(self.url,proxies=self.proxyDict) print g.status_code print g.content print self.url stemps['init']['end'] = time.time() #print stemps['init']['end'] - stemps['init']['start'] x= stemps['init']['end'] - stemps['init']['start'] print x test=BaseCheck(url='http://google.com') test.getrequests() 

Answer 6

This is a bit late, but here is a wrapper class that makes it easier to slip proxies and then does http POST or GET:

Proxyrequests

 https://github.com/rootVIII/proxy_requests 

Answer 7

If you want to persist cookies and session data, the best thing to do is:

 import requests proxies = { 'http': 'http://user:pass@10.10.1.0:3128', 'https': 'https://user:pass@10.10.1.0:3128', } # Create the session and set the proxies. s = requests.Session() s.proxies = proxies # Make the HTTP request through the session. r = s.get('http://www.showmemyip.com/')