I am currently debugging a random error with our applications. I will try to provide as many details as possible. Basically, after some time, one of the application pools for the application on our server stops. I have to start it, or sometimes restart Coldfusion or, even worse, for everything to work again.
Server logs indicate that the pool was stopped due to a quick restore operation in IIS. (The application executes X number of errors in a short amount of time, so IIS stops the application)
By looking at the ColdFusion error logs, I get this immediately during an IIS error, where it stops my pool.
SEVERE: error in getRealPathFromConn java.net.SocketException: Connection reset by peer: error writing socket on java.net.SocketOutputStream.socketWrite0 (native method) on java.net.SocketOutputStream.socketWrite (SocketOutputStream.java:92) on java.net .SocketOutputStream.write (SocketOutputStream.java:124) on org.apache.coyote.ajp.AjpProcessor.getRealPathFromServer (AjpProcessor.javahaps91) ... AND SO ON
If you really need a complete track, I can provide it, but it is long. I'm not sure what getRealPathFromConn is. I tried Googling but haven't come up with a conclusion yet. In the traces, they lead to different files in different applications, but these lines have nothing to do with each other.
Any thoughts on what could be the reason or where I should look further will help the ton.
EDIT: I forgot to say, I “believe” that it started on Monday. The only thing that changed before Monday was the following.
We have an application structure:
root/app1/application.cfc root/app2/application.cfc root/app3/application.cfc root/app3/website1 root/app3/website2 root/app3/website3
I changed the structure, so the two websites have their own application.cfc, because we are slowly deploying changes in the operation of websites. While I'm testing this on two of them. Since I made this change, an application that continues to split the application pool for application 1. None of the sites, but I feel it could be connected.
UPDATE Updated Coldfusion to the latest version. (10,0,7,283649). The systems worked for about a week and crashed this morning.
Coldfusion-error.log at this time:
Jan 25, 2013 8:09:11 AM org.apache.catalina.core.AprLifecycleListener init INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: C:\\ColdFusion10\\cfusion\lib;C:\\ColdFusion10\\cfusion\jintegra\bin;C:\\ColdFusion10\\cfus ion\jintegra\bin\international;C:\\ColdFusion10\\cfusion\lib\oosdk\classes\win Jan 25, 2013 8:09:12 AM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["ajp-bio-8012"] Jan 25, 2013 8:09:12 AM org.apache.catalina.core.StandardService startInternal INFO: Starting service Catalina Jan 25, 2013 8:09:12 AM org.apache.catalina.core.StandardEngine startInternal INFO: Starting Servlet Engine: Apache Tomcat/7.0.23 Jan 25, 2013 8:09:21 AM org.apache.catalina.core.ApplicationContext log INFO: ColdFusionStartUpServlet: ColdFusion: Starting application services Jan 25, 2013 8:09:21 AM org.apache.catalina.core.ApplicationContext log INFO: ColdFusionStartUpServlet: ColdFusion: VM version = 20.4-b02 Jan 25, 2013 8:09:29 AM org.apache.catalina.core.ApplicationContext log INFO: CFMxmlServlet: Macromedia Flex Build: 87315.134646 Jan 25, 2013 8:09:35 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler ["ajp-bio-8012"] Jan 25, 2013 8:09:35 AM com.adobe.coldfusion.launcher.Launcher run INFO: Server startup in 25362 ms Jan 25, 2013 8:10:04 AM org.apache.coyote.ajp.AjpProcessor getRealPathFromServer SEVERE: Error in getRealPathFromConn java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:168) at org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:319) at org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:445) at org.apache.coyote.ajp.AjpProcessor.readString(AjpProcessor.java:425) at org.apache.coyote.ajp.AjpProcessor.getRealPathFromServer(AjpProcessor.java:407) at org.apache.coyote.ajp.AbstractAjpProcessor.getRealPath(AbstractAjpProcessor.java:1154) at org.apache.naming.resources.FileDirContext.doGetRealPath(FileDirContext.java:198) at org.apache.naming.resources.BaseDirContext.getRealPath(BaseDirContext.java:412) at org.apache.catalina.core.StandardContext.getRealPath(StandardContext.java:4475) at org.apache.catalina.core.ApplicationContext.getRealPath(ApplicationContext.java:414) at org.apache.catalina.core.ApplicationContextFacade.getRealPath(ApplicationContextFacade.java:335) at coldfusion.runtime.ServletContextWrapper.doGetRealPath(ServletContextWrapper.java:179) at coldfusion.runtime.ServletContextWrapper._doGetRealPath(ServletContextWrapper.java:116) at coldfusion.runtime.ServletContextWrapper.getRealPath(ServletContextWrapper.java:98) at coldfusion.runtime.TemplateProxyFactory.getFullName(TemplateProxyFactory.java:1092) at coldfusion.runtime.TemplateProxyFactory.resolveName(TemplateProxyFactory.java:202) at coldfusion.runtime.TemplateProxyFactory.resolveName(TemplateProxyFactory.java:159) at coldfusion.runtime.TemplateProxyFactory.resolveFile(TemplateProxyFactory.java:120) at coldfusion.cfc.CFCProxy.<init>(CFCProxy.java:138) at coldfusion.cfc.CFCProxy.<init>(CFCProxy.java:84) at coldfusion.runtime.AppEventInvoker.<init>(AppEventInvoker.java:64) at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:226) at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48) at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40) at coldfusion.filter.PathFilter.invoke(PathFilter.java:112) at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:94) at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28) at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38) at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46) at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38) at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22) at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62) at coldfusion.CfmServlet.service(CfmServlet.java:219) at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42) at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:414) at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:204) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:298) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)
The GetRealPathFromConn exception throws about 20 or more times in a row. In the case of Viewer, I have, of course, 5 warnings right before IIS disables the pool:
A process serving application pool 'domain.com' terminated unexpectedly. The process id was '3732'. The process exit code was '0xff'. A process serving application pool 'domain.com' suffered a fatal communication error with the Windows Process Activation Service. The process id was '3052'. The data field contains the error number. A process serving application pool 'domain.com' terminated unexpectedly. The process id was '7972'. The process exit code was '0xff'. A process serving application pool 'domain.com' suffered a fatal communication error with the Windows Process Activation Service. The process id was '4736'. The data field contains the error number. A process serving application pool 'domain.com' terminated unexpectedly. The process id was '8016'. The process exit code was '0xff'. Application pool 'domain.com' is being automatically disabled due to a series of failures in the process(es) serving that application pool.