Segmentation error using newtonsoft json on linux with mono

Question

Running Mono 3.2.6 on Debian 7

When using Newtonsoft.Json to deserialize objects in a multi-threaded application, my application crashes due to a “segmentation error”.

I attached gdb and here is the output:

root@ns38225 :/home/stress# gdb program 30380 GNU gdb (GDB) 7.4.1-debian Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... program: No such file or directory. Attaching to process 30380 Reading symbols from /opt/mono-3.0/bin/mono-sgen...done. Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6 Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/librt.so.1 Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2 Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread 0x798bb10bb700 (LWP 30381)] Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0 Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib/x86_64-linux-gnu/libnss_compat.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libnss_compat.so.2 Reading symbols from /lib/x86_64-linux-gnu/libnsl.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libnsl.so.1 Reading symbols from /lib/x86_64-linux-gnu/libnss_nis.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libnss_nis.so.2 Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2 0x0000798bb363f8ad in nanosleep () from /lib/x86_64-linux-gnu/libpthread.so.0 Mono support loaded. (gdb) handle SIGXCPU SIG33 SIG35 SIGPWR nostop noprint Signal Stop Print Pass to program Description SIGXCPU No No Yes CPU time limit exceeded SIGPWR No No Yes Power fail/restart SIG33 No No Yes Real-time event 33 SIG35 No No Yes Real-time event 35 (gdb) continue Continuing. [New Thread 0x798bb1105700 (LWP 30432)] [New Thread 0x798bb0cd4700 (LWP 30433)] [New Thread 0x798bade0f700 (LWP 30434)] [New Thread 0x798ba64a2700 (LWP 30435)] [New Thread 0x798ba5461700 (LWP 30436)] [New Thread 0x798bae3cd700 (LWP 30437)] [Thread 0x798ba64a2700 (LWP 30435) exited] [New Thread 0x798ba64a2700 (LWP 30438)] [Thread 0x798bae3cd700 (LWP 30437) exited] [New Thread 0x798bae3cd700 (LWP 30439)] [Thread 0x798ba5461700 (LWP 30436) exited] Program received signal SIGSEGV, Segmentation fault. copy_object_no_checks ( obj=obj@entry =0x798bb2f35c40, queue=queue@entry =0x96a200) at sgen-copy-object.h:110 110 gboolean has_references = SGEN_VTABLE_HAS_REFERENCES (vt); (gdb) thread apply all backtrace Thread 10 (Thread 0x798bae3cd700 (LWP 30439)): #0 0x0000798bb32d9824 in sigsuspend () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00000000005c73e4 in suspend_thread (info=0x212c4d0, context=context@entry = 0x798bae3cb5c0) at sgen-os-posix.c:120 #2 0x00000000005c755b in suspend_handler (sig=<optimized out>, siginfo=<optimized out>, context=0x798bae3cb5c0) at sgen-os-posix.c:148 #3 <signal handler called> #4 0x00000000005ceca6 in sgen_ptr_in_nursery (p=0x798bb2fc03e8) at ../../mono/metadata/sgen-gc.h:306 #5 mono_gc_wbarrier_arrayref_copy (dest_ptr=0x798bb2fc03e8, src_ptr=0x798bb2ffac50, count= 33) at sgen-gc.c:4318 #6 0x0000000000536d94 in ves_icall_System_Array_FastCopy (length=<optimized out>, dest_idx=0, dest=<optimized out>, source_idx=0, source=0x798bb2ffac30) at icall.c:782 #7 ves_icall_System_Array_FastCopy (source=0x798bb2ffac30, source_idx=0, dest=<optimized out>, dest_idx=0, length=<optimized out>) at icall.c:692 #8 0x0000000041ecfcf6 in ?? () #9 0x000000000215fff0 in ?? () #10 0x00000000020b9bd8 in ?? () #11 0x00000000020b9bd8 in ?? () #12 0x00000000005e652b in sgen_fragment_allocator_par_alloc (allocator=0x798bae3cbd30, size=133640910515248) at sgen-nursery-allocator.c:475 #13 0x0000000000000000 in ?? () Thread 9 (Thread 0x798ba64a2700 (LWP 30438)): #0 0x0000798bb32d9824 in sigsuspend () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00000000005c73e4 in suspend_thread (info=0x215f900, context=context@entry = 0x798ba64a09c0) at sgen-os-posix.c:120 #2 0x00000000005c755b in suspend_handler (sig=<optimized out>, siginfo=<optimized out>, context=0x798ba64a09c0) at sgen-os-posix.c:148 #3 <signal handler called> #4 0x0000000041e1ece4 in ?? () #5 0x0000798bb2fb3908 in ?? () #6 0x0000000040fa7185 in ?? () #7 0x0000000000000004 in ?? () #8 0x0000798bb2fb3ac8 in ?? () #9 0x0000798bb2ff70a0 in ?? () #10 0x0000798bb2ff70a0 in ?? () #11 0x0000000000000022 in ?? () #12 0x0000000000000022 in ?? () #13 0x0000798bb2fb3908 in ?? () #14 0x0000000040fadcb0 in ?? () #15 0x0000798bb2fb3ac8 in ?? () #16 0x0000798bb2ff70a0 in ?? () ---Type <return> to continue, or q <return> to quit--- #17 0x0000798bb2fb3908 in ?? () #18 0x0000000040fadc34 in ?? () #19 0x0000798bb2fb3ac8 in ?? () #20 0x0000798bb2ff70a0 in ?? () #21 0x0000798bb2fb3908 in ?? () #22 0x0000000040fadbc8 in ?? () #23 0x0000000000000004 in ?? () #24 0x0000798bb2fb3908 in ?? () #25 0x0000798bb2ff70a0 in ?? () #26 0x0000000040fad420 in ?? () #27 0x0000798bb2fb3908 in ?? () #28 0x0000798ba64a1130 in ?? () #29 0x0000000000000000 in ?? () Thread 5 (Thread 0x798bade0f700 (LWP 30434)): #0 0x0000798bb363ecec in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000798bb363a339 in _L_lock_926 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x0000798bb363a15b in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0 #3 0x00000000005d23a2 in mono_gc_set_skip_thread (skip=0) at sgen-gc.c:5605 #4 0x00000000005889a0 in async_invoke_thread (data=0x0, data@entry =0x9694e0) at threadpool.c:1573 #5 0x0000000000584082 in start_wrapper_internal (data=0x2116100) at threads.c:608 #6 start_wrapper (data=0x2116100) at threads.c:653 #7 0x0000000000617971 in thread_start_routine ( args=args@entry =0x1de5cb0) at wthreads.c:294 #8 0x0000000000627530 in inner_start_thread (arg=0x211bcf0) at mono-threads-posix.c:49 #9 0x0000798bb3637b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #10 0x0000798bb3381a7d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #11 0x0000000000000000 in ?? () Thread 4 (Thread 0x798bb0cd4700 (LWP 30433)): #0 0x0000798bb363ecec in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000798bb363a339 in _L_lock_926 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x0000798bb363a15b in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0 #3 0x00000000005d23a2 in mono_gc_set_skip_thread (skip=0) at sgen-gc.c:5605 #4 0x00000000005889a0 in async_invoke_thread (data=0x0, data@entry =0x9694e0) at threadpool.c:1573 #5 0x0000000000584082 in start_wrapper_internal (data=0x1f50ca0) at threads.c:608 #6 start_wrapper (data=0x1f50ca0) at threads.c:653 #7 0x0000000000617971 in thread_start_routine ( args=args@entry =0x1de5be8) at wthreads.c:294 #8 0x0000000000627530 in inner_start_thread (arg=0x1f25520) at mono-threads-posix.c:49 #9 0x0000798bb3637b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #10 0x0000798bb3381a7d in clone () from /lib/x86_64-linux-gnu/libc.so.6 ---Type <return> to continue, or q <return> to quit--- #11 0x0000000000000000 in ?? () Thread 3 (Thread 0x798bb1105700 (LWP 30432)): #0 0x0000798bb32d9824 in sigsuspend () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00000000005c73e4 in suspend_thread (info=0x1f4edf0, context=context@entry = 0x798bb1104780) at sgen-os-posix.c:120 #2 0x00000000005c755b in suspend_handler (sig=<optimized out>, siginfo=<optimized out>, context=0x798bb1104780) at sgen-os-posix.c:148 #3 <signal handler called> #4 0x0000798bb363f8ad in nanosleep () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x0000000000616c57 in SleepEx ( ms=ms@entry =500, alertable=alertable@entry =1) at wthreads.c:842 #6 0x0000000000585d83 in monitor_thread ( unused=unused@entry =0x0) at threadpool.c:779 #7 0x0000000000584082 in start_wrapper_internal (data=0x1f4e6e0) at threads.c:608 #8 start_wrapper (data=0x1f4e6e0) at threads.c:653 #9 0x0000000000617971 in thread_start_routine ( args=args@entry =0x1de5b20) at wthreads.c:294 #10 0x0000000000627530 in inner_start_thread (arg=0x1f25770) at mono-threads-posix.c:49 #11 0x0000798bb3637b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #12 0x0000798bb3381a7d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #13 0x0000000000000000 in ?? () Thread 2 (Thread 0x798bb10bb700 (LWP 30381)): #0 0x0000798bb32d9824 in sigsuspend () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00000000005c73e4 in suspend_thread (info=0x1e687a0, context=context@entry = 0x798bb10ba7c0) at sgen-os-posix.c:120 #2 0x00000000005c755b in suspend_handler (sig=<optimized out>, siginfo=<optimized out>, context=0x798bb10ba7c0) at sgen-os-posix.c:148 #3 <signal handler called> #4 0x0000798bb363e41e in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x0000000000622d68 in mono_sem_wait ( sem=sem@entry =0x9698a0, alertable=alertable@entry = 1) at mono-semaphore.c:116 #6 0x00000000005a01b5 in finalizer_thread ( unused=unused@entry =0x0) at gc.c:1073 #7 0x0000000000584082 in start_wrapper_internal (data=0x1e69370) at threads.c:608 #8 start_wrapper (data=0x1e69370) at threads.c:653 #9 0x0000000000617971 in thread_start_routine ( args=args@entry =0x1de5800) at wthreads.c:294 #10 0x0000000000627530 in inner_start_thread (arg=0x1e69890) at mono-threads-posix.c:49 #11 0x0000798bb3637b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #12 0x0000798bb3381a7d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #13 0x0000000000000000 in ?? () Thread 1 (Thread 0x798bb40ec780 (LWP 30380)): #0 copy_object_no_checks ( obj=obj@entry =0x798bb2f35c40, queue=queue@entry =0x96a200) ---Type <return> to continue, or q <return> to quit--- at sgen-copy-object.h:110 #1 0x00000000005ef921 in simple_nursery_serial_copy_object_from_obj (queue=0x96a200, obj_slot=0x798ba5941e80) at sgen-minor-copy-object.h:206 #2 simple_nursery_serial_scan_object (start=<optimized out>, queue=0x96a200) at sgen-scan-object.h:78 #3 0x00000000005cbc77 in sgen_drain_gray_stack ( max_objs=max_objs@entry =-1, ctx=...) at sgen-gc.c:1192 #4 0x00000000005d1576 in collect_nursery (finish_up_concurrent_mark=0, unpin_queue=0x0) at sgen-gc.c:2611 #5 collect_nursery (unpin_queue=0x0, finish_up_concurrent_mark=0) at sgen-gc.c:2483 #6 0x00000000005d1b08 in sgen_perform_collection (requested_size=4080, generation_to_collect=0, reason=<optimized out>, wait_to_finish=<optimized out>) at sgen-gc.c:3445 #7 0x00000000005e7bf4 in mono_gc_alloc_obj_nolock ( vtable=vtable@entry =vtable("string"), size=size@entry =4080) at sgen-alloc.c:264 #8 0x00000000005e7dbb in mono_gc_alloc_string (vtable=vtable("string"), size=4080, len= 2024) at sgen-alloc.c:563 #9 0x00000000408397e0 in ?? () #10 0x0000000001e5ef60 in ?? () #11 0x0000798baf5f3550 in ?? () #12 0x0000798ba4fa4ad8 in ?? () #13 0x0000798ba5d3c268 in ?? () #14 0x0000798bb2fd8688 in ?? () #15 0x0000000000000ff0 in ?? () #16 0x00007ffffe9284d0 in ?? () #17 0x0000798bb2fd7698 in ?? () #18 0x0000798bb40ec758 in ?? () #19 0x0000000001e4c198 in ?? () #20 0x00000000000007e8 in ?? () #21 0x0000000040831394 in ?? () #22 0x0000798baff90018 in ?? () #23 0x0000798bb2fd7378 in ?? () #24 0x0000798bb2cdd1a8 in ?? () #25 0x0000798baf58f810 in ?? () #26 0x0000000000000000 in ?? () 

Here is the code to reproduce the problem (remember the hard data directory)

 using System; using System.Collections.Generic; using System.IO; using System.Linq; using System.Text; using System.Threading; using System.Threading.Tasks; using Newtonsoft.Json; namespace TaskStressTest { class IntObj { public int Int; } class Program { static long highestId = 0; private static long done = 0; static object cntLock = new object(); static List<IntObj> tIds = new List<IntObj>(); static void Main(string[] args) { Thread.Sleep(60000); var filePaths = Directory.GetFiles("/home/cs/data", "*.json"); var fileCount = filePaths.Length; var f = 0; foreach (var filePath in filePaths) { var tasks = new List<Task>(); var file = File.OpenRead(filePath); var stream = new StreamReader(file); var line = ""; while (!String.IsNullOrWhiteSpace(line = stream.ReadLine())) { var line1 = line; tasks.Add(Task.Run(async() => { var tId = new IntObj {Int = Thread.CurrentThread.ManagedThreadId}; lock (cntLock) { tIds.Add(tId); var ints = tIds.Select(x => x.Int).Distinct().OrderBy(i1 => i1).ToList(); var active = ints.Count(); if (active > highestId) { highestId = active; Console.WriteLine(active); } } await DoWork(line1); lock (cntLock) { tIds.Remove(tId); } })); } tasks.ForEach(task => { task.Wait(); var doneNow = ++done; //Console.WriteLine("D1 - " + doneNow); }); } Console.WriteLine("Done"); } static async Task DoWork(string line) { try { JsonConvert.DeserializeObject(line); } catch (Exception) { } } } } 

Actually, I have no idea where to go from here, can anyone give me any instructions on what to do next?

UPDATE

I just changed my code to delegate deserialization to a loop in the main thread, and the nickname happened again, this time in a little more detail:

 Stacktrace: at <unknown> <0xffffffff> at (wrapper managed-to-native) object.__icall_wrapper_mono_gc_alloc_vector (intptr,intptr,intptr) <0xffffffff> at (wrapper alloc) object.AllocVector (intptr,intptr) <0xffffffff> at Newtonsoft.Json.JsonTextReader..ctor (System.IO.TextReader) <0x0006b> at Newtonsoft.Json.JsonConvert.DeserializeObject (string,System.Type,Newtonsoft.Json.JsonSerializerSettings) <0x000af> at Newtonsoft.Json.JsonConvert.DeserializeObject (string) <0x00013> at TaskStressTest.Program/<>c__DisplayClass28.<DoWork>b__26 () <0x0001b> at TaskStressTest.Program.Main (string[]) <0x00134> at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object (object,intptr,intptr,intptr) <0xffffffff> 

UPDATE 2

Another error has occurred.

 * Assertion: should not be reached at sgen-scan-object.h:111 Stacktrace: at <unknown> <0xffffffff> at (wrapper managed-to-native) object.__icall_wrapper_mono_gc_alloc_vector (intptr,intptr,intptr) <0xffffffff> at (wrapper alloc) object.AllocVector (intptr,intptr) <0xffffffff> at System.Array.Resize<T> (T[]&,int) <0x000d3> at System.Collections.Generic.List`1.set_Capacity (int) <0x00053> at System.Collections.Generic.List`1.GrowIfNeeded (int) <0x0006b> at System.Collections.Generic.List`1.Insert (int,T) <0x00047> at Newtonsoft.Json.Linq.JContainer.InsertItem (int,Newtonsoft.Json.Linq.JToken,bool) <0x0022e> at Newtonsoft.Json.Linq.JProperty.InsertItem (int,Newtonsoft.Json.Linq.JToken,bool) <0x00033> at Newtonsoft.Json.Linq.JContainer.AddInternal (int,object,bool) <0x002a6> at Newtonsoft.Json.Linq.JContainer.Add (object) <0x0003f> at Newtonsoft.Json.Linq.JTokenWriter.AddValue (Newtonsoft.Json.Linq.JValue,Newtonsoft.Json.JsonToken) <0x0003c> at Newtonsoft.Json.Linq.JTokenWriter.AddValue (object,Newtonsoft.Json.JsonToken) <0x00043> at Newtonsoft.Json.Linq.JTokenWriter.WriteValue (string) <0x0003f> at Newtonsoft.Json.JsonWriter.WriteToken (Newtonsoft.Json.JsonReader,int,bool,bool) <0x003bd> at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateJObject (Newtonsoft.Json.JsonReader) <0x000ab> at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject (Newtonsoft.Json.JsonReader,System.Type,Newtonsoft.Json.Serialization.JsonContract,Newtonsoft.Json.Serialization.JsonProperty,Newtonsoft.Json.Serialization.JsonContainerContract,Newtonsoft.Json.Serialization.JsonProperty,object) <0x000e7> at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal (Newtonsoft.Json.JsonReader,System.Type,Newtonsoft.Json.Serialization.JsonContract,Newtonsoft.Json.Serialization.JsonProperty,Newtonsoft.Json.Serialization.JsonContainerContract,Newtonsoft.Json.Serialization.JsonProperty,object) <0x000bf> at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize (Newtonsoft.Json.JsonReader,System.Type,bool) <0x001c7> at Newtonsoft.Json.JsonSerializer.DeserializeInternal (Newtonsoft.Json.JsonReader,System.Type) <0x00487> at Newtonsoft.Json.JsonSerializer.Deserialize (Newtonsoft.Json.JsonReader,System.Type) <0x0001b> at Newtonsoft.Json.JsonConvert.DeserializeObject (string,System.Type,Newtonsoft.Json.JsonSerializerSettings) <0x000d7> at Newtonsoft.Json.JsonConvert.DeserializeObject (string) <0x00013> at TaskStressTest.Program/<>c__DisplayClass28.<DoWork>b__26 () <0x0001b> at TaskStressTest.Program.Main (string[]) <0x0013c> at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object (object,intptr,intptr,intptr) <0xffffffff> Native stacktrace: mono() [0x4ad6a1] /lib/x86_64-linux-gnu/libpthread.so.0(+0xf030) [0x694d59a22030] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x694d596bb475] /lib/x86_64-linux-gnu/libc.so.6(abort+0x180) [0x694d596be6f0] mono() [0x62bcbd] mono() [0x62bdf6] mono() [0x5ef457] mono() [0x5cbc77] mono() [0x5d1576] mono() [0x5d1b08] mono() [0x5e7cba] mono() [0x5e7ecb] [0x4086d270] 

UPDATE 3

When using DataContractJsonSerializer instead of Json.NET everything works fine, so there is almost certainly a problem with this library when used in mono (because it works fine on windows without mono) .del>

Ignoring this, this again happened with the DataContractJsonSerializer , this time no useful stack traces were found.

UPDATE 4

I also tried running the application under Mono on Windows (both Json.NET and DataContractJsonSerializer ), I usually get an Access violation reading location exception, but I also saw Assertion: should not be reached at sgen-scan-object.h:111 Application works fine if I do not run it under mono; /

I also tried to get rid of all the features of .NET 4.5 and switch to runtime 4.0 using Task.Factory.StartNew and remove all async / wait, but the problem Task.Factory.StartNew . Now I am trying to use other deserialization libraries.