NetConnection.Call.Failed happens sporadically in Flex3 / Tomcat / BlazeDS / Spring

Question

I have a very big problem. I wrote a great application using Flex3 / Tomcat / BlazeDS / Spring that worked very well when developing locally, great when I was deployed in a common dev environment, but then very often did not work when deployed in our test environment.

Errors seem to occur most often when a remote access request takes a fairly long time (more than 20 seconds). Errors occur on my dev server, but only when the request takes a very long time (more than 45 seconds). However, the error also occurs, apparently, instantly at times (send a request, and it will work immediately). Most errors also have an HTTP status: 502 on them (Bad Gateway), but some just say that HTTP Failed.

I did not do anything special with BlazeDS, except to put the war file in the webapps directory. Access to the application is via https. Remote calls refer to the server only at the "destination". The tomcat servers on our DEV and TEST instances have the same Java arguments (same Xms and Xmx, etc.). Below are my relevant files:

services-config.xml

<?xml version="1.0" encoding="UTF-8"?> <services-config> <services> <default-channels> <channel ref="my-secure-amf"/> </default-channels> </services> <channels> <channel-definition id="my-secure-amf" class="mx.messaging.channels.SecureAMFChannel"> <endpoint url="https://{server.name}:{server.port}/{context.root}/messagebroker/amfsecure" class="flex.messaging.endpoints.SecureAMFEndpoint"/> <properties> <add-no-cache-headers>false</add-no-cache-headers> </properties> </channel-definition> </channels> <services-config> 

web.xml

 <?xml version="1.0" encoding="UTF-8"?> <web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"> ... <servlet> <servlet-name>spring-flex</servlet-name> <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class> <init-param> <param-name>contextConfigLocation</param-name> <param-value>/WEB-INF/flexContext.xml</param-value> </init-param> <load-on-startup>1</load-on-startup> </servlet> <servlet-mapping> <servlet-name>spring-flex</servlet-name> <url-pattern>/messagebroker/*</url-pattern> </servlet-mapping> </web-app> 

flexContext.xml

 <!-- I have component-scan tags here to scan my class files. The Controller I'm using for flex has an annotation on it to define it as a remote destination. Here it is: @Controller @RemotingDestination(value = "dest", channels="my-secure-amf") public class FlexController { --> <bean id="flexExceptionTranslator" class="edu.liberty.zconduct.web.FlexExceptionTranslator" /> <flex:message-broker> <flex:exception-translator ref="flexExceptionTranslator"/> <!-- <flex:secured /> Had this previously, but it wasn't working then, either --> </flex:message-broker> 

I'm already close to the end of my rope. The error is a breaker application. Please help me if you can.

EDIT I switched to http and am still getting an error. Tomcat now says:

WARNING: Error sending final packet

java.net.SocketException: Broken pipe

I look in my apache log for jk_mod and see some debugging information that looks like the request I sent, and then the following:

[debugging] ajp_send_request :: jk_ajp_common.c (1592): leave the request body (432)

[debugging] jk_shutdown_socket :: jk_connect.c (681): About the closed socket 68

[debugging] jk_shutdown_socket :: jk_connect.c (732): turn off socket 68 and read 0 long bytes

[Information] ajp_connection_tcp_get_message :: jk_ajp_common.c (1150): (myTestServer) cannot receive response header message from tomcat, network problems or tomcat (xx.xx.xx.xx: xxxx) does not work (errno = 11)

[error] ajp_get_reply :: jk_ajp_common.c (1962): (myTestServer) Tomcat is down or refused to connect. No response has been sent to the client (yet)

[info] ajp_service :: jk_ajp_common.c (2447): (myTestServer) sending a request to tomcat failed (restored), (Attempt = 2)

[error] ajp_service :: jk_ajp_common.c (2466): (myTestServer) connecting tomcat failed.

[debugging] ajp_reset_endpoint :: jk_ajp_common.c (743): (myTestServer) reset endpoint with sd = 4294967295 (socket shutdown)

[debug] ajp_done :: jk_ajp_common.c (2905): reuse pool slot = 0 for working myTestServer

[info] jk_handler :: mod_jk.c (2615): Maintenance error = 0 for Worker = myTestServer

Answer 1

Cornel Creanga helped me come to this decision. I switched my application to http and started receiving messages in the catalina.out file. I do not know why this influenced it, but it was so. The reports said that there was a broken pipe in the socket. Further research showed that jk_mod was involved.

Due to the large number of search queries, I found that our SysAdmins set the socket timeout on the server to 10 seconds on our TEST servers, but 5 minutes on our DEV servers, so mass searches made it time out when it received a TEST.

SysAdmins configured it in TEST so that another worker (with a 5-minute timeout) handles calls for this context path.