Geek Answers Handbook

Python urllib2: cannot assign the requested address

I send thousands of requests using urllib2 with a proxy. I got a lot of errors while doing:

 urlopen error [Errno 99] Cannot assign requested address

I read here that this could be due to a connection that is already connected. This is true? Any suggestions on how to fix this?

0
source share
2 answers

Here is the answer to a similar question that I prepared earlier .... much earlier ... Socket usage error when reusing sockets

The error is different, but the main problem is probably the same: you consume all available ports and try to reuse them before the TIME_WAIT state has ended.

[EDIT: in response to comments]

If this is the application / specification for your application, one obvious strategy is to control the connection speed to avoid this situation.

Alternatively, you can use the httplib module. httplib.HTTPConnection() allows you to specify the source_address tuple, with which you can specify the port from which you can connect, for example. this will connect to localhost: 1234 from localhost: 9999:

 import httplib conn = httplib.HTTPConnection('localhost:1234', source_address=('localhost',9999)) conn.request('GET', '/index.html')

Then it is a matter of controlling the assignment of the source port, as described in my earlier answer. If you are on Windows, you can use this method to bypass the default port range of 1024-5000.

There is (of course) an upper limit on how many connections you are going to make, and it is doubtful which application would require a quick connection to thousands of connections.

+4
source

As mhawke suggested, the TIME_WAIT question seems most likely. A system solution for your situation may be to configure kernel parameters so that such connections are cleared more often. Two options:

 $ sysctl net.ipv4.tcp_tw_recycle=1

This will allow the kernel to reuse connections in TIME_WAIT state. This can cause problems with NAT settings. Other:

 $ sysctl net.ipv4.tcp_max_orphans=8192 $ sysctl net.ipv4.tcp_orphan_retries=1

This suggests that the kernel supports no more than 8192 connections that are not connected to any user process, and only once tries to restart TCP connections.

Please note that these are not permanent changes. Add the setting to /etc/sysctl.conf to make them permanent.

http://code.google.com/p/lusca-cache/issues/detail?id=89#c4
http://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.kernel.obscure.html

+1
source

More articles:


All Articles