WinDbg: hunting exceptions that caused .net service to crash

Question

You are having problems with a short-term failure of the xnet service on the application server. The service can work for several hours, days or weeks without any problems, but then fall with a little information.

The service starts in a cluster (3 x services on each server) on two servers - and failures were detected in any of the services on both servers. The replicated environment shows the same behavior, so I have "exhausted" the idea of ​​a configuration problem.

Initial error from the application server event log:

Error message from event log on server XXXX Application: MySvc.exe Framework Version: v4.0.30319 Description: The process was terminated due to an internal error in the .NET Runtime at IP 000007FEEFD8CD4C (000007FEEFC70000) with exit code 80131506 

It doesn’t show much detail, and the best signposts I have found on the Internet is the “crosshair” ...

Application crash with "Internal Error in .NET Runtime"

http://www.jamesewelch.com/2010/09/30/troubleshooting-internal-error-in-the-net-runtime/

Finally, after a month of working with the attached AdPlus debugger, we have a crash line and some crash dumps. Now that I have landfills, I am having problems getting anything useful from them.

I have already studied several freezing dumps with great success several times and read Tess Ferrandes' blogs a lot, but the dumps that I experience are dead. Most objects, exceptions, etc. All are marked for garbage collection, and only the main thread remains - I probably missed something.

I will add details from! analysis -v, as well as dump logs - which show exceptions.

So - the real question is this: can someone give me some directions where to go from here. The exceptions in the dump log do not match what I see on the actual dump.

DUMP 1 Register here: http://pastebin.com/Eg5YCqww

DUMP 1 Analysis: (I have a problem with characters that I cannot solve ..)

 0:000> !analyze -v *** FAULTING_IP: +112c9440 00000000`00000000 ?? ??? EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff) ExceptionAddress: 0000000000000000 ExceptionCode: 80000003 (Break instruction exception) ExceptionFlags: 00000000 NumberParameters: 0 FAULTING_THREAD: 00000000000011f8 PROCESS_NAME: MySvc.exe ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION} Breakpoint A breakpoint has been reached. EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) - One or more arguments are invalid MOD_LIST: <ANALYSIS/> NTGLOBALFLAG: 0 APPLICATION_VERIFIER_FLAGS: 0 MANAGED_STACK: (TransitionMU) 000000000022EBB0 000007FEF40CB1AB System_ServiceProcess_ni!DomainBoundILStubClass.IL_STUB_PInvoke(IntPtr)+0x3b 000000000022EC70 000007FEF40CD20D System_ServiceProcess_ni!System.ServiceProcess.ServiceBase.Run(System.ServiceProcess.ServiceBase[])+0x26d 000000000022EDA0 000007FF00170227 MySvc!Ax.Remoting.MySvc.Main()+0x107 (TransitionUM) MANAGED_STACK_COMMAND: _EFN_StackTrace BUGCHECK_STR: APPLICATION_FAULT_WRONG_SYMBOLS_FILL_PATTERN_ffffffff PRIMARY_PROBLEM_CLASS: WRONG_SYMBOLS_FILL_PATTERN_ffffffff DEFAULT_BUCKET_ID: WRONG_SYMBOLS_FILL_PATTERN_ffffffff LAST_CONTROL_TRANSFER: from 000007fefd8810ac to 000000007760f6fa STACK_TEXT: 00000000`0022e818 000007fe`fd8810ac : 00000000`007541f0 000007fe`f40ce089 00000000`0022e9c0 00000000`00000000 : ntdll!ZwWaitForSingleObject+0xa 00000000`0022e820 000007fe`fe7daffb : 00000000`ffffffff 000007fe`fe7d344c 00000000`00000000 00000000`0000032c : KERNELBASE!WaitForSingleObjectEx+0x79 00000000`0022e8c0 000007fe`fe7d9d61 : 00000000`01d47ff0 00000000`0000032c 00000000`00000000 00000000`00000000 : sechost!ScSendResponseReceiveControls+0x13b 00000000`0022e9b0 000007fe`fe7d9c16 : 00000000`0022eb18 00000000`00000000 00000000`00000000 000007fe`00000000 : sechost!ScDispatcherLoop+0x121 00000000`0022eac0 000007fe`f19017c7 : 00000000`11213890 00000000`01d635c0 00000000`00000000 00000000`00000000 : sechost!StartServiceCtrlDispatcherW+0x14e 00000000`0022eb10 000007fe`f40cb1ab : 00000000`01d63680 00000000`0022ebe8 000007fe`f40a5b50 0000bf6c`4589127e : clr!DoNDirectCall__PatchGetThreadCall+0x7b 00000000`0022ebb0 000007fe`f40cd20d : 00000000`01d63680 00000000`00000000 00000000`01d63698 00000000`00000000 : System_ServiceProcess_ni+0x2b1ab 00000000`0022ec70 000007ff`00170227 : 00000000`10ff1ac8 00000000`10ff1af0 00000000`10ff1af0 00000000`10ff1af0 : System_ServiceProcess_ni+0x2d20d 00000000`0022eda0 000007fe`f196dc54 : 00000000`0022ee80 000007fe`f1904e65 ffffffff`fffffffe 00000000`0022f3a0 : 0x7ff`00170227 00000000`0022ee30 000007fe`f196dd69 : 000007ff`000551f8 00000000`00000001 00000000`00000000 00000000`00000000 : clr!CallDescrWorker+0x84 00000000`0022ee70 000007fe`f196dde5 : 00000000`0022ef88 00000000`00000000 00000000`0022ef90 00000000`0022f168 : clr!CallDescrWorkerWithHandler+0xa9 00000000`0022eef0 000007fe`f1a214c5 : 00000000`00000000 00000000`0022f178 00000000`00000000 00000000`00000000 : clr!MethodDesc::CallDescr+0x2a1 00000000`0022f120 000007fe`f1a215fc : 00000000`000ad7c0 00000000`000ad7c0 00000000`00000000 00000000`00000000 : clr!ClassLoader::RunMain+0x228 00000000`0022f370 000007fe`f1a213b2 : 00000000`0022f970 00000000`00000200 00000000`000b7a80 00000000`00000200 : clr!Assembly::ExecuteMainMethod+0xac 00000000`0022f620 000007fe`f1ac6d66 : 00000000`00000000 00000000`10fd0000 00000000`00000000 00000000`00000000 : clr!SystemDomain::ExecuteMainMethod+0x452 00000000`0022fbd0 000007fe`f1ac6c83 : 00000000`10fd0000 00000000`00000000 00000000`00000000 00000000`00000000 : clr!ExecuteEXE+0x43 00000000`0022fc30 000007fe`f1a2c515 : 00000000`000ad7c0 ffffffff`ffffffff 00000000`00000000 00000000`00000000 : clr!CorExeMainInternal+0xc4 00000000`0022fca0 000007fe`f8973309 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`0022fc88 : clr!CorExeMain+0x15 00000000`0022fce0 000007fe`f8a05b21 : 000007fe`f1a2c500 000007fe`f89732c0 00000000`00000000 00000000`00000000 : mscoreei!CorExeMain+0x41 00000000`0022fd10 00000000`773bf56d : 000007fe`f8970000 00000000`00000000 00000000`00000000 00000000`00000000 : mscoree!CorExeMain_Exported+0x57 00000000`0022fd40 00000000`775f2cc1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd 00000000`0022fd70 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d STACK_COMMAND: ~0s; .ecxr ; kb FOLLOWUP_IP: sechost!ScSendResponseReceiveControls+13b 000007fe`fe7daffb 85c0 test eax,eax SYMBOL_STACK_INDEX: 2 SYMBOL_NAME: sechost!ScSendResponseReceiveControls+13b FOLLOWUP_NAME: MachineOwner MODULE_NAME: sechost IMAGE_NAME: sechost.dll DEBUG_FLR_IMAGE_TIMESTAMP: 4a5be05e FAILURE_BUCKET_ID: WRONG_SYMBOLS_FILL_PATTERN_ffffffff_80000003_sechost.dll!ScSendResponseReceiveControls BUCKET_ID: X64_APPLICATION_FAULT_WRONG_SYMBOLS_FILL_PATTERN_ffffffff_sechost!ScSendResponseReceiveControls+13b 

UPDATE 1 (December 29):

One of the CLR exceptions from the dump log is restored, then the call stack follows. It seems like an exception is raised when db is called (via ODAC)

 clr!RaiseTheExceptionInternalOnly+0x363 clr!IL_Throw+0x146 gm.a(System.String, System.String, Int32, System.String, XXBase, Int32, XXDataParameter[]) gm.b(XXBase, XXBase, Boolean, Boolean, Boolean, Int32) gm.b(XXBase, XXBase) od.a(XXGridQueue, TaskStatus, ProcessResult, Int32, Int32, Int32) od.b(XXGridQueue) he.b(XXBaseCollection) he.a(Boolean ByRef) XX.MySvc.tmr_Elapsed(System.Object) System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) 

Reconstructed access violation exception stacks. The error is caused when the garbage collector is called after the ODAC library is called.

 (1330.1074): Access violation - code c0000005 (first chance) FirstChance_av_AccessViolation clr!WKS::gc_heap::plan_phase+0x5ac clr!WKS::gc_heap::gc1+0xbb clr!WKS::gc_heap::garbage_collect+0x276 clr!WKS::GCHeap::GarbageCollectGeneration+0x14e clr!WKS::gc_heap::try_allocate_more_space+0x25f clr!WKS::GCHeap::Alloc+0x7e clr!FastAllocatePrimitiveArray+0xc5 clr!JIT_NewArr1+0x389 System.Decimal.GetBits(System.Decimal) Oracle.DataAccess.Types.DecimalConv.GetDecimal(IntPtr) Oracle.DataAccess.Client.OracleDataReader.GetDecimal(Int32) Oracle.DataAccess.Client.OracleDataReader.GetValue(Int32) Oracle.DataAccess.Client.OracleDataReader.GetValues(System.Object[]) jr.a(System.Data.IDataReader, Boolean, ku, Boolean, DbTypeEnum, System.Type[]) ls.a(System.Data.IDataReader, Boolean, ku, Boolean, DbTypeEnum, System.Type[]) ba.a(System.String, System.Data.IDataReader, Boolean, ku, Boolean, System.Type[]) ... XX.MySvc.tmr_Elapsed(System.Object) 

Possible similar problems (from new information): http://markmail.org/message/yy3mvbngula4i3mu#query:+page:1+mid:l546gn5sfxtxxm5i+state:results http://social.msdn.microsoft.com/Forums/en / clr / thread / 33920b39-690c-42c8-b04a-0f1f7176835a

UPDATE 2 (February 23):

ODAC components were upgraded to the correct version of Dotnet 4.0 (or listed as compatible on the Oracle website), and the problem still repeated. It is still repeated very intermittently, every week or one week. The service that happens is cyclically changing every day.

You have some dumps from the most recent failures, and they still indicate heap corruption - although not complete dumps (access violation). In fact, it seems that creating a complete dump failed.

 Creating d:\dumps\2xx_Crash_Mode\FULLDUMP_FirstChance_epr_Process_Shut_Down_MySvc.exe__0344.dmp - mini user dump WriteFullMemory.Memory.Read(0x262c000, 0x1000) failed 0x8007012b, ABORT. Dump creation failed, Win32 error 0n299 "Only part of a ReadProcessMemory or WriteProcessMemory request was completed." 

In addition, a user-managed (dotnet) library is loaded into the application, and this also seems to throw an exception, although this is just a “first chance” and does not seem to cause the process to crash (I assume this could be a factor). This is also our library, so I can check if it invoked managed code. Mistake:

 EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff) ExceptionAddress: 000007fefcffaa7d (KERNELBASE!RaiseException+0x0000000000000039) ExceptionCode: c0000006 (In-page I/O error) ExceptionFlags: 00000000 NumberParameters: 3 Parameter[0]: 0000000000000000 Parameter[1]: 000000006d34aca0 Parameter[2]: 00000000c00000c4 Inpage operation failed at 000000006d34aca0, due to I/O error 00000000c00000c4 PROCESS_NAME: MySvc.exe ERROR_CODE: (NTSTATUS) 0xc0000006 - The instruction at 0x%p referenced memory at 0x%p. The required data was not placed into memory because of an I/O error status of 0x%x. EXCEPTION_OBJECT: !pe 1a8106a8 Exception object: 000000001a8106a8 Exception type: System.Runtime.InteropServices.SEHException Message: External component has thrown an exception. InnerException: <none> StackTrace (generated): SP IP Function 000000002C77B980 0000000000000000 ... 000000002C77BA50 000007FF01DCBA51 ... StackTraceString: <none> HResult: 80004005 MANAGED_OBJECT: !dumpobj 148306f8 Name: System.String MethodTable: 000007feed9a6870 EEClass: 000007feed52ed58 Size: 112(0x70) bytes File: C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll String: External component has thrown an exception. Fields: MT Field Offset Type VT Attr Value Name 0000000000000000 4000103 8 System.Int32 1 instance 43 m_stringLength 0000000000000000 4000104 c System.Char 1 instance 45 m_firstChar 000007feed9a6870 4000105 10 System.String 0 shared static Empty >> Domain:Value 00000000002a69f0:NotInit 000000000dd738d0:NotInit << EXCEPTION_MESSAGE: External component has thrown an exception. MANAGED_OBJECT_NAME: System.Runtime.InteropServices.SEHException MANAGED_STACK_COMMAND: !pe 1a8106a8 LAST_CONTROL_TRANSFER: from 000007fef47e8fc1 to 000007fefcffaa7d ADDITIONAL_DEBUG_TEXT: Followup set based on attribute [Is_ChosenCrashFollowupThread] from Frame:[0] on thread:[PSEUDO_THREAD] ; Followup set based on attribute [ip_is_call_value_Arch_si] from Frame:[23] on thread:[162c] FAULTING_THREAD: ffffffffffffffff BUGCHECK_STR: APPLICATION_FAULT__SYSTEM.RUNTIME.INTEROPSERVICES.SEHEXCEPTION_APPLICATION_FAULT_CALL PRIMARY_PROBLEM_CLASS: _SYSTEM.RUNTIME.INTEROPSERVICES.SEHEXCEPTION_CALL DEFAULT_BUCKET_ID: _SYSTEM.RUNTIME.INTEROPSERVICES.SEHEXCEPTION_CALL STACK_TEXT: 00000000`2c77b980 00000000`00000000 ... 00000000`2c77ba50 00000000`ffffffff ... 

Anyone with any ideas on how to pursue this further in an expedient manner. I wish you to get more complete dumps - but, of course, you need to find the answers earlier than the next failure!

Answer 1

The reason for your failure (by hitting a break point) is the presence of a heap in the process. Heap corruption errors are reported using heap management functions by issuing a debug interrupt.

Judging by the reported errors, the .net runtime is not ready to handle them (I may be wrong, and there may be a better explanation). A common way to track heap faults is to have a (full) page heap that helps locate the offending component, breaking the process closer to the point of damage.

Hunting a bunch of corruption is a real pain, to say the least, but if memory consumption allows it, I would be most effective for applications with moderate memory requirements with a full bunch of pages.

Hope this helps.

Answer 2

GC x64.NET 4.0 has an error. Perhaps it affected you. MS recommends disabling the parallel GC until it receives a fix. Alternatively, you can enable the GC server to receive one GC thread per core, which is possible if you have multiple cores.

Otherwise, the gc server flag will have no effect.

Here is a link to a KB article .

Answer 3

A couple of things 1. Make sure you are using the latest version of clr 2. For your heap, corruption is a good option, and you can try GCStress for management. How to enable GCStress in Windows 7? 3. To check heap corruption on a managed heap, you can use checkheap, which is part of SOS https://msdn.microsoft.com/en-us/library/bb190764(v=vs.110).aspx "VerifyHeap Checks the collector heap garbage for signs of damage and displays all errors found. Heap corruption can be caused by calling platforms that are created incorrectly. "