Apache ConnectionClosedException: connection closed, HttpAsyncRequestExecutor.endOfInput (HttpAsyncRequestExecutor.java data44)

Question

we have three applications A, B and C deployed on the same tomcat server. There is an HTTP call that occurs between AB (REST CALL) and another HTTP call from B to C (REST CALL)

At first we used a synchronous HTTP call, recently we changed our code to an asynchronous call. We used HttpAsyncClients for (application A to invoke application B) and for (application B to invoke application C).

we get an intermittent org.apache.http.ConnectionClosedException: For some time we get this exception in the logs of application A (Async Client to B) and some time in application B (Async Client to C)

CloseableHttpAsyncClient client = HttpAsyncClients.custom() .setMaxConnPerRoute(100)) .setMaxConnTotal(config.getInt(150)).build(); 

and for each request we gave time

 final RequestConfig params = RequestConfig.custom().setConnectTimeout(300000) // 5min .setSocketTimeout(300000).setConnectionRequestTimeout(300000).build(); 

Find the stack trace below:

  org.apache.http.ConnectionClosedException: Connection closed at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.endOfInput(HttpAsyncRequestExecutor.java:344) [httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:261) [httpcore-nio-4.4.5.jar:na] at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) [httpasyncclient-4.1.2.jar:na] at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) [httpasyncclient-4.1.2.jar:na] at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) [httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) [httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) [httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) [httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) [httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) [httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) [httpcore-nio-4.4.5.jar:4.4.5] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] 

Note that this question is not duplicated by an Intermittent ConnectionClosedException with httpasynclient stacj tracing is completely different. And this event happens intermittently.