Socket.read () gap between JBoss and ActiveMQ

Question

The

• My Java application is a WAR deployed to JBoss (4.0.4GA)
• Publishes and subscribes to an ActiveMQ instance (5.6.0)
• Java application uses Apache Camel (2.10.3) for all integration (creation and consumption) using ActiveMQ
• JBoss and ActiveMQ themselves (CentOS 5.6 Final) are quad-core virtual servers, each virtual is on a different physical

I have a thread hang problem and I see the following in a thread dump:

java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.read(SocketInputStream.java:129) java.io.BufferedInputStream.fill(BufferedInputStream.java:218) java.io.BufferedInputStream.read1(BufferedInputStream.java:258) java.io.BufferedInputStream.read(BufferedInputStream.java:317) sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687) sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632) sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195) java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379) org.springframework.remoting.httpinvoker.SimpleHttpInvokerRequestExecutor.validateResponse(SimpleHttpInvokerRequestExecutor.java:146) org.springframework.remoting.httpinvoker.SimpleHttpInvokerRequestExecutor.doExecuteRequest(SimpleHttpInvokerRequestExecutor.java:66) org.springframework.remoting.httpinvoker.AbstractHttpInvokerRequestExecutor.executeRequest(AbstractHttpInvokerRequestExecutor.java:136) org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:192) org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:174) org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:142) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) $Proxy117.SigmaCruxer(Unknown Source) com.tms.SigmaClient.SigmaClient.processMessage(SigmaClient.java:46) com.tms.SigmaClient.SigmaServiceEndpoint.doSigma(SigmaServiceEndpoint.java:29) com.tms.SigmaClient.SigmaServiceEndpoint.mark(SigmaServiceEndpoint.java:43) sun.reflect.GeneratedMethodAccessor193.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:597) org.apache.camel.component.bean.MethodInfo.invoke(MethodInfo.java:329) org.apache.camel.component.bean.MethodInfo$1.proceed(MethodInfo.java:231) org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:169) org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:104) org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:74) org.apache.camel.impl.ProcessorEndpoint.onExchange(ProcessorEndpoint.java:102) org.apache.camel.impl.ProcessorEndpoint$1.process(ProcessorEndpoint.java:72) org.apache.camel.impl.converter.AsyncProcessorTypeConverter$ProcessorToAsyncProcessorBridge.process(AsyncProcessorTypeConverter.java:50) org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:78) org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:114) org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:284) org.apache.camel.processor.SendProcessor.process(SendProcessor.java:109) org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:78) org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98) org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89) org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:69) org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:78) org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98) org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89) org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:99) org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:78) org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:318) org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:209) org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:305) org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:102) org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:78) org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98) org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89) org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:69) org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:104) org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:85) org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:91) org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:560) org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:498) org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467) org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325) org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263) org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058) org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050) org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) java.lang.Thread.run(Thread.java:662) 

In accordance with these two skills: ( here and here ), my JBoss application has an I / O lock on Socket.read() , awaiting completion of the response from the service provider located below (in my case, ActiveMQ). Again, according to these articles, the culprit is one of the following 3 points:

• ActiveMQ is in an unhealthy / unstable state and reacts too slowly, as a result of which the listen / wait / lock threads hang; or
• The ActiveMQ instance itself works fine, but it handles an operation (writing to KahaDB, etc.) that ends too long, again causing my threads to hang; or
• There are network issues between my JBoss application (WAR) and my ActiveMQ instance.

I am trying to figure out which of the three takes place. Is there anything in this thread dump to indicate which one it is? My understanding (after reading these articles) is that the real hang is that the client-side socket (blocking) has just not received all the bytes it needs to consider completing the response; that is, he did not receive any response from ActiveMQ or simply did not receive a full response.

Therefore, I ask:

• Is there a clear idea of ​​which of the three scenarios takes place? If so, what / why? If not, what should be my next step (I am also the “administrator” who configured ActiveMQ so that I have full access to it, as well as JBoss and WAR deployed on it).
• Can a JBoss update upgrade to this? Perhaps 4.0.4GA uses the "old" (blocking) Java I / O, whereas newer versions can use NIO? Probably a long shot, but he still cannot discredit him.
• Both articles emphasize that the correct socket-timeout configuration should be implemented, which can greatly mitigate all this (although it does not address the problems associated with abandoning ActiveMQ and / or networks):
  • Is this a timeout that I would write in my Java code? If so, how and with which API? Jms Some banners on the client side ActiveMQ?
  • Is this a timeout that I am implementing at the OS level? If so, I'm not sure how to proceed ...
  • Is this a timeout that I implement on the server side (ActiveMQ)? If so, how?

I think I'm closing the solution here, but it seems to be stuck and hard to see the forest through the trees. Thanks in advance!

Answer 1

I have some experience with JBoss (and Glassfish) and ActiveMQ, but I have never used Camel before (but I am familiar with the Mule that I read, it looks like).

The stack trace looks like this: Camel is trying to associate ActiveMQ (JMS material at the bottom of the trace) with the website endpoint (HTTP material over the trace).

I am a little confused as to where Camel (CamelContext) works. You said that you have two virtual machines: one for JBoss and one for ActiveMQ. In my case, we run Mule ESB on a machine with our ActiveMQ. Where does your camel work?

Your stack trace is more like task number 1 from the first message. It’s as if the Camel part cannot “see” the endpoint of the Internet. Ensure that your WAR is deployed correctly and that your web endpoint (WSDL) is visible from both virtual machines. Check endpoints; perhaps one is installed on localhost or something else that would not allow it to switch to another machine.

It is a little difficult to say if it is incomplete reading or complete inability to read. Is any data passing through? Perhaps the web server is slowly overloading and cannot keep up with requests (and some threads are starving, as in your error). Socket timeouts become important when you have slow responses or many requests; if you can create a simple test (quick and with few queries), you can at least verify that you have basic connectivity. What data entry (test) caused this error?

I will be happy to try to improve this answer, given the greater contribution. (Sorry, I would try to comment on your question, but I don't think I have a reputation for this yet ...)