How to create many (I mean many) sockets on Linux?

Question

I tried to create a simple program in different languages ​​(C #, Java, C ++, PHP) to connect to the server, and they all behave the same way. Therefore, I believe that this problem is more related to the OS level.

Basically, I want the program to connect to the server using a TCP socket and send 1 byte, and then close the socket. This must be done thousands of times per second and maintained over a period of time. This is for server benchmarking.

So far, after several thousand client sockets, the system has completely stopped. It can only start creating sockets in a minute or so. I made sure that I closed every socket after the transfer.

Now this problem is familiar with servers such as Apache, where utilities (like ab / siege) recommend testing Apache using the keep-alive protocol. Ie, create a small number of TCP connections, but make a few requests with them for testing purposes. However, this is not possible in our case, since our own server does not support HTTP and does not support the HTTP 1.1 support model.

So how can this be achieved? I checked the following limitations

• ulimit installed a very large number
• TCP TIME_WAIT fixed by setting /proc/sys/net/ipv4/tcp_rw_recycle and /proc/sys/net/ipv4/tcp_rw_reuse to 1. (I really confirmed that netstat no TIME_WAIT sockets)
• This is not due to restrictions on the number of threads / processes. I tried to restart my client application, and this is the same. When the OS refuses new sockets, nothing will help.

PS. This is NOT a server side restriction. We tested this by buying another box and running the same client code when the first client box refused to create new sockets. The server did a great job with this. We do not want to buy 5-10 boxes and rotate between them to overcome this problem.

OS: Fedora 10 Linux 2.6.24-23-xen # 1 SMP

Answer 1

An old joke: A man goes to the doctor, says: "Doctor, it hurts me when I do this," twisting my arm into a strange position.

The doctor replies: "Well, do not do this!"

See what you do, this is a very unnatural process. Establishing a TCP connection requires a handshake that transmits bytes far exceeding one byte per message. Installation time and downtime will be significant. It is very likely that what you are doing uses the core resources associated with this handshake; Of course, if you then leave him alone and stop clapping him, he will eventually catch up.

So what are you really trying to measure? What are you really trying to do? If you are really trying to send one byte at a time - the gods forbid - at least consider using udp; there is no terrible installation / breakdown. It is still extremely inefficient compared to overhead - even a UDP packet requires something like 20 bytes of framing - but it is better.

Answer 2

Take a look at an article by Richard Jones, The Millionth Comet App with Mochiweb, Part 3 . This is about deploying the Comet application in Erlang, but the section "Enabling it to 1 million" describes how he compared his server; it opens with the statement "Creating a million tcp connections from one host is not trivial." This should give you some idea of ​​what you need.

Answer 3

Have you tried setting the SO_REUSEADDR flag on the socket?

Answer 4

Is it possible that you have run out of ports? You only get 5000 - 1024 ports if you don't want to call bind () in a loop to find the next free port.

bind () with 0 for the port returns a free port in the range 1024-5000. bind () with the specified port receives this port, if available.

 int bindnextport(int s, struct sockaddr sa) { static int nextport = 1025; int lastport; lastport = nextport; do { sa.sa_data[0] = nextport >> 8; sa.sa_data[1] = nextport & 255; if (!bind(s, &sa, sizeof(sa)) return 0; ++nextport; if (nextport >= 65536) nextport = 1024; } while (lastport != nextport); return 1; } 

Answer 5

Connecting and then sending 1 byte is not a standard, except perhaps for the TCP protocol itself. As Charlie Martin said, most of the time is wasted and then disconnects the socket.

I understand that you WANT to test, but is this really a good idea of ​​what your application is doing? Are you really going to configure the connection to send only 1 byte?

Answer 6

The nginx http server claims to be capable of supporting 10,000 inactive HTTP-keep-alive connections. You can see how they do it .