Why does performance drop dramatically when my Erlang TCP proxy receives many concurrent requests?

Question

For training purposes, I wrote a simple TCP proxy in Erlang. It works, but I experience an odd performance drop when I use ab (Apache Bench) to make many simultaneous requests. This is not a decline in productivity in itself, which makes me wonder, but the extent of the decline. The backend is nginx as a web server. My proxy is between ab and nginx.

This is the code for my proxy.

-module(proxy). -export([start/3]). start(InPort, OutHost, OutPort) -> {ok, Listen} = gen_tcp:listen(InPort, [binary, {packet, 0}, {active, once}]), spawn(fun() -> connect(Listen, OutHost, OutPort) end). connect(Listen, OutHost, OutPort) -> {ok, Client} = gen_tcp:accept(Listen), spawn(fun() -> connect(Listen, OutHost, OutPort) end), {ok, Server} = gen_tcp:connect(OutHost, OutPort, [binary, {packet, 0}, {active, once}]), loop(Client, Server). loop(Client, Server) -> receive {tcp, Client, Data} -> gen_tcp:send(Server, Data), inet:setopts(Client, [{active, once}]), loop(Client, Server); {tcp, Server, Data} -> gen_tcp:send(Client, Data), inet:setopts(Server, [{active, once}]), loop(Client, Server); {tcp_closed, _} -> ok end. 

Running 64 consecutive requests in my proxy server. I get a very good result.

 ab -n 64 127.0.0.1:80/ Concurrency Level: 1 Time taken for tests: 0.097 seconds Complete requests: 64 Failed requests: 0 Write errors: 0 Total transferred: 23168 bytes HTML transferred: 9664 bytes Requests per second: 659.79 [#/sec] (mean) Time per request: 1.516 [ms] (mean) Time per request: 1.516 [ms] (mean, across all concurrent requests) Transfer rate: 233.25 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.3 0 1 Processing: 1 1 0.5 1 2 Waiting: 0 1 0.4 1 2 Total: 1 1 0.5 1 2 Percentage of the requests served within a certain time (ms) 50% 1 66% 2 75% 2 80% 2 90% 2 95% 2 98% 2 99% 2 100% 2 (longest request) 

This is a bit slower than using Apache Bench directly against nginx.

But, firing 64 concurrent requests in the proxy, performance drops insanely

 ab -n 64 -c 64 127.0.0.1:80/ Concurrency Level: 64 Time taken for tests: 2.011 seconds Complete requests: 64 Failed requests: 0 Write errors: 0 Total transferred: 23168 bytes HTML transferred: 9664 bytes Requests per second: 31.82 [#/sec] (mean) Time per request: 2011.000 [ms] (mean) Time per request: 31.422 [ms] (mean, across all concurrent requests) Transfer rate: 11.25 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 31 121.7 0 501 Processing: 3 1135 714.4 1001 2006 Waiting: 3 1134 714.3 1000 2005 Total: 3 1167 707.8 1001 2006 Percentage of the requests served within a certain time (ms) 50% 1001 66% 1502 75% 2003 80% 2004 90% 2005 95% 2005 98% 2005 99% 2006 100% 2006 (longest request) 

What / where is the problem? I expected lower performance, but why so much? Look at queries per second!

It does not seem to be that important. I give erl many threads using + A. I even tried SMP, but the results are almost the same.

My setup: Windows 7 64, Intel QuadCore, 8 GB RAM. I get similar results on Ubuntu using 128 concurrent requests.

EDIT: A new understanding is included. The total number of requests does not matter. This is just the number of simultaneous requests.

Answer 1

This connect/3 is serial:

 connect(Listen, OutHost, OutPort) -> {ok, Client} = gen_tcp:accept(Listen), spawn(fun() -> connect(Listen, OutHost, OutPort) end), 

You cannot accept a new connection until the new process for creating gen_tcp:accept/1 is ready. This may be due to a bottleneck in your code. You can try the "acceptor" pool to improve performance in this case. I will also try adding catch all clause to loop/2 receive to avoid accidentally stuffing the mailbox.

And also what are your erl options? Are +A streams turned on and +K true ?

Answer 2

Have you tried the same tests directly with nginx? if it is not configured correctly, it may also have the same performance drop.

Answer 3

I can not reproduce your results. I tried your tests using apache, yaws and nginx as web servers and found very few variations that ran with the proxy server and without it with any of them. I ran them on Linux, so maybe this is a problem with Windows or the erlang version of the Windows virtual machine.

Answer 4

Have you tried to increase the lag of the listening socket with the {backlog, B} gen_tcp:listen option?

The default value is 5, I think you should try with a higher value, like 30 or maybe 50.