The spray client throws an “Too many open files” exception when providing more concurrent requests

Question

I have a client-client for spray that runs on server X that will connect to server Y. Server Y is slow (it takes 3 + seconds for the request)

This is the call to my http client:

def get() { val result = for { response <- IO(Http).ask(HttpRequest(GET,Uri(getUri(msg)),headers)).mapTo[HttpResponse] } yield response result onComplete { case Success(res) => sendSuccess(res) case Failure(error) => sendError(res) } } 

These are the configurations that I have in application.conf:

 spray.can { client { request-timeout = 30s response-chunk-aggregation-limit = 0 max-connections = 50 warn-on-illegal-headers = off } host-connector { max-connections = 128 idle-timeout = 3s } } 

Now I tried to abuse server X with a lot of concurrent requests (using ab with n = 1000 and c = 100).

Up to 900 requests everything went fine. After that, the server threw many exceptions, and after that I could not get to the server. These are exceptions:

[info] [ERROR] [03/28/2015 17: 33: 13.276] [squbs-akka.actor.default-dispatcher-6] [akka: // squbs / system / IO-TCP / selectors / $ a / 0 ] Acceptance Error: Failed to accept new connection

[info] java.io.IOException: too many open files [info] at sun.nio.ch.ServerSocketChannelImpl.accept0 (native method) [info] at sun.nio.ch.ServerSocketChannelImpl.accept (ServerSocketChannelImpl.java:241) [info] at akka.io.TcpListener.acceptAllPending (TcpListener.scala: 103)

and upon further access to the same server, he chose the following exception:

[info] [ERROR] [03/28/2015 17: 53: 16.735] [hcp-client-akka.actor.default-dispatcher-6] [akka: // hcp-client / system / IO-TCP / selectors] null [info] akka.actor.ActorInitializationException: throw exception

[info] at akka.actor.ActorInitializationException $ .apply (Actor.scala: 164)

[info] at akka.actor.ActorCell.create (ActorCell.scala: 596)

[info] Called: java.lang.reflect.InvocationTargetException

[info] at sun.reflect.GeneratedConstructorAccessor59.newInstance (Unknown source)

[info] Caused by: java.io.IOException: Too many open files [info] at sun.nio.ch.IOUtil.makePipe (native method)

I previously used the apache http client (which was synchronous), which was able to handle 10,000+ requests with a concurrency of 100.

I'm not sure I missed something. Any help would be appreciated.

Answer 1

The problem is that every time you call the get() method, it creates a new actor that creates at least one connection to the remote server. In addition, you never close this actor, so each such connection leaves before it expires.

You need only one such actor to manage all of your HTTP requests, so IO(Http) from the get() method is required to fix it and call it only once. Reuse that returned ActorRef for all your requests to this server. Close it when the application terminates.

For example:

 val system: ActorSystem = ... val io = IO(Http)(system) io ! Http.Bind( ... def get(): Unit = { ... io.ask ... // or io.tell ... }