How to avoid a NoRouteToHostException?

Question

Disclosure: The code I'm working on is a term paper at a university.

Reference Information. The task I am trying to complete is to report on the impact of various stream processing methods. To do this, I wrote several classes that respond to a request from a client using Java Sockets. The idea is to load the server with requests and report how various thread strategies handle this. Each client will make 100 requests, and at each iteration we increase the number of clients by 50 until something breaks.

Problem: An exception repeatedly and consistently occurs:

 Caused by: java.net.NoRouteToHostException: Cannot assign requested address
     at java.net.PlainSocketImpl.socketConnect (Native Method)
     at java.net.PlainSocketImpl.doConnect (PlainSocketImpl.javahaps33)

This occurs in several scenarios, including when the client and server are running on the local host. Connections may be successful for a while, and shortly after trying to connect 150 clients that throw an exception.

My first thought was that this is a Linux limitation for open file descriptors (1024), but I don't think so. I also verified that all connections between sockets are closed properly (i.e. in the correct finally block).

I hesitate to publish the code, because I’m not sure which parts will be most relevant and don’t want to have a huge list of code in the question.

Has anyone come across this before? How can I avoid a NoRouteToHostException?

---

EDIT (additional questions in italics)

There are still good answers that point to either the Ephemeral Port Range or RFC 2780. Both of them suggest that I have too many connections. For both, he shows that the number of connections that must be made to reach this limit suggests that at some point I am not closing the connections.

After debugging both the client and the server, both watched how the call to the myJava-Net-SocketInstance.close() method hit. This assumes that the connections are closed (at least in the non-exclusive case). Is this the right offer?

In addition, is there an OS level needed to resume access to ports? It would be possible to run the program separately for each 50+ clients, if the next attempt just requires a short period (or optimistic, running the command).

---

EDIT v2.0

Having received good answers, I changed my code to use the setReuseAddress (true) method with every socket connection made on the client. This did not have the desired effect, and I am still limited to 250-300 clients. When the program finishes executing the netstat -a command, there are many socket connections in the TIME_WAIT status.

My assumption was that if the socket was in TIME-WAIT state and was installed with the SO-REUSEADDR , any new sockets trying to use this port might - however, I still get a NoRouteToHostException.

It is right? Is there anything else that can be done to solve this problem?

Answer 1

Have you tried setting:

 echo "1" >/proc/sys/net/ipv4/tcp_tw_reuse 

and / or

 echo "1" >/proc/sys/net/ipv4/tcp_tw_recycle 

These settings may force Linux to reuse TIME_WAIT sockets. Unfortunately, I can not find the final documentation.

Answer 2

This can help:

Ephemeral Port Range

Another important ephemeral range of ports is that it limits the maximum number of connections from one machine for a particular service to a remote machine! TCP / IP uses a 4-tuple connection to distinguish between connections, so if the ephemeral range of the ports is only 4,000 ports, which means that there are only 4,000 unique connections from the client machine to the remote one.

So you may have run out of available ports. For the number of available ports, see

 $ cat /proc/sys/net/ipv4/ip_local_port_range 32768 61000 

Quit my Ubuntu system, where I would have 28,232 ports for client connections. Therefore, your test will fail as soon as you have 280+ clients.

Answer 3

The requested address cannot be assigned - this is the error string for the EADDRNOTAVAIL error.

I suspect you have run out of source ports. In the dynamic range, 16,383 sockets are available, available for use as the source port (see RFC 2780 ). 150 clients * 100 connections = 15,000 ports - so you are likely to click this limit.

Answer 4

If you have run out of source ports, but actually they do not support many open connections, set the SO_REUSEADDR option. This will allow you to reuse local ports that are still in TIME_WAIT state.

Answer 5

If you close 500 connections per second, you will not have sockets. If you connect to the same locations (web servers) that use keepalive, you can implement connection pools, so you do not close or open sockets.

It will also save the processor.

Using tcp_tw_recycle and tcp_tw_reuse can cause packets to come from the previous connection, so there is a need to wait 1 minute to clear the packets.

Answer 6

For any other Java users who stumble over this issue, I would recommend using the connection pool so that the connections are used correctly.