IIS7: Invalid w3wp.exe application, what is the cause of these crashes?

Question

Our site is in .NET, but with some old ASP and 32bits libraries. It worked fine for a while (2 years). But over the past month, we saw the following error on our IIS7 server, which we could not track and fix:

"W3wp.exe application is incorrect, version 7.0.6001.18000, time stamp 0x47919413, kernel32.dll error module, version 6.0.6001.18215, time stamp 0x4995344f, exception code 0xe053534f, error offset 0x0002f328, process ID 0x% 9, application start time 0x% 10."

We can reproduce the error:

• One of our .ASPX pages starts loading, executing code and requests (we have response.flush () throughout the page to track code breaks), then it suddenly stops, and we get the above error in IIS.

• The page stops loading, and without response.flush () it is not redirected to our error.aspx page (as indicated in web.config)

• The error does NOT occur all the time. Sometimes this happens 3 times in a row, then it works fine for 15 minutes without stopping with the correct redirect to error.aspx.

• The error we get then is a classic: "Either BOF, or EOF - True, or the current record has been deleted."

• If an error occurs, the page freezes, and all other sessions on the same computer from any browser also have hanging web pages (BTW, we allow only one workflow during testing). From other computers, the site loads fine.

• I can recycle the application pool, kill w3wp.exe, restart IIS. That's that. The only way to successfully load the page again is to restart MS SQL, which processes our session states. I don’t know why this is so, but we guessed that session cookies in user browsers indicate a thread that was not completed properly (due to the aforementioned failure), and IIS is waiting for it to complete to process more code (?). If someone can explain this better, it will be really helpful. Is there a timeout we can set to “complete” threads? Is this a problem related to MS SQL?

I also considered using private and virtual memory, because I believe that our code is not the most efficient, and I am sure that we still have memory leaks. However, I saw the page crash, although private and virtual memories were still pretty low (up to 100 MB each).

I used Debug Diag and WinDbg as indicated here: http://blogs.msdn.com/b/tess/archive/2009/03/20/debugging-a-net-crash-with-rules-in-debug-diag .aspx , but we cannot do windbg work, that is what we are trying to do at the moment.

If someone can help us or point us in the right direction, which would be really great, thanks.

Answer 1

“Either BOF or EOF is True, or the current record has been deleted” means the table is empty and you are trying to make MoveNext. So, check that you make any movements .

IIS is known for throwing kernel errors in w3wp.exe, like this one. All your mistakes in the state of the session are only symptoms of a crashed process. Several APP pools will not help - they just propagate the error.

I bet this is SQL locks due to a change in your user environment. This will cause a 10 second lag as SQL tries to determine which query to kill. One wins, one loses. The player returns a pointer to an unexpectedly empty table, and you are trying to perform a transition and subsequent failure. Perhaps you can point your database to the ODBC connection and enable tracing, or figure out a way to force SQL to register it.

I had all the same symptoms as Perl. I was able to wrap fn () to execute all SQL queries and register all sql, + parameters and any errors on the disk to identify the problem. These were dead ends, then we were able to program automatic repetition, and in the end we re-encoded the order of the queries and the scanned columns to eliminate deadlocks.

Answer 2

It is possible that one of your referenced / related assemblies was accidentally corrupted (may happen) on disk somewhere. Can you try to repeat the problem on a new, clean machine with the same statistics, fresh installations of the latest xyz drivers that you use?

I solved a mysterious problem that took me months to isolate this path. It seemed that clean, new machines with the same specifications and pre-drivers would work very well - only some old machines with the same characteristics failed sequentially. As a result, I deleted everything (IIS, ASP.NET, .NET, database and client) and started from scratch. The final reason, when I isolated it, was that the db client driver was damaged on old machines (and all old machines were clones of each other, so I assume they were cloned after corruption occurred), and it seems , it fiddled with the .NET memory space, even when I did not call it directly. I have not yet answered my "help me debug this monster" with this answer because I doubted it would ever help anyone.

Answer 3

We started getting this error after installing Windows updates on a Windows Server 2008R2 machine. The Windows Process Activation Service (WAS) installs some additional site bindings that cause problems for our installation.

We removed the bindings net.tcp, net.pipe, net.msmq and msmq.formatname from our site and no longer received an exception due to an error.

Answer 4

This is probably an edge case, but in case someone comes here and they use MVCMailer , I got the same error due to the .SendAsync () method in email programs.

I switched them all to .Send () and the failure stopped.

See this SO answer to use an asynchronous mailbox and avoid a crash (presumably I haven't implemented it personally)