High memory issues in .NET Framework 4 but not in 4.5

Question

I have the following code snippet (.net 4) that consumes a lot of memory:

struct Data { private readonly List<Dictionary<string,string>> _list; public Data(List<Dictionary<string,string>> List) { _list = List; } public void DoWork() { int num = 0; foreach (Dictionary<string, string> d in _list) { foreach (KeyValuePair<string, string> kvp in d) num += Convert.ToInt32(kvp.Value); } Console.Write(num); //_list = null; } } class Test1 { BlockingCollection<Data> collection = new BlockingCollection<Data>(10); Thread th; public Test1() { th = new Thread(Work); th.Start(); } public void Read() { List<Dictionary<string, string>> l = new List<Dictionary<string, string>>(); Random r = new Random(); for (int i=0; i<100000; i++) { Dictionary<string, string> d = new Dictionary<string,string>(); d["1"] = r.Next().ToString(); d["2"] = r.Next().ToString(); d["3"] = r.Next().ToString(); d["4"] = r.Next().ToString(); l.Add(d); } collection.Add(new Data(l)); } private void Work() { while (true) { collection.Take().DoWork(); } } } class Program { Test1 t = new Test1(); static void Main(string[] args) { Program p = new Program(); for (int i = 0; i < 1000; i++) { ptRead(); } } } 

The size of the blocking collection is 10. As far as I know, gc should collect links in the data structure as soon as its DoWork method is finished. Nevertheless, the memory continues to grow rapidly until the program crashes or crashes on its own, and this happens more often on low-speed machines (on some machines the memory does not increase). Also, when I add the following line "_list = null;" at the end of the DoWork method and convert the "data" to a class (from structure), memory does not increase.

What could be here? I need some suggestions here.

Update: The problem occurs on computers with .NET Framework 4 installed (4.5 not )

Answer 1

If you read Steven Tuub's description of how ConcurrentQueue works, the behavior makes sense. BlockingCollection uses ConcurrentQueue by default, which stores its items in linked lists from 32-element segments.

For the purpose of simultaneous access, the elements in the linked list are never overwritten, so they are not obtained without a link until the last of the entire segment 32 is destroyed. Since you have a limited capacity of 10 elements, let's say that you created 41 elements and consumed 31. This means that you will have one segment of the 31 consumed items plus one item in the queue, and another segment with the remaining 9 items. At this point, all 41 items are referenced, so if each item is 25 MB, your collection will take up 1 GB! As soon as the next item is consumed, all 32 elements in the head segment will not be found and can be collected.

You might think that there should be only 10 elements in the queue, and this will be the case for a non-competitive queue, but this will not allow one thread to list the elements in the queue, while the other thread has been producing or consuming elements.

The reason the .Net 4.5 infrastructure is not leaking is because they changed the behavior to null elements as soon as they were created until no one lists the queue. If you start listing the collection , you should see a memory leak even within .Net 4.5.

The reason the _list = null option works when you have a class is because you create a “box” wrapper that allows you not to reference the list in every place where it was used. Setting a value in your local variable changes the same copy as in the queue.

The reason why setting _list = null does not work when you have a struct , you can only modify copies of the struct . The "original" version of seating in this segment of the queue is virtually unchanged because ConcurrentQueue does not provide a way to change it. In other words, you only change the copy of the value in your local variable, and not copy the copy to the queue.

Answer 2

I tried on my computer here is the result:

• With Data as a class and without _list = null at the end of DoWork -> memory increases
• With Data as a structure and without _list = null at the end of DoWork -> memory increases
• With Data as a class and with _list = null at the end of DoWork -> memory is stabilized at a speed of 150 MB
• With Data as a struct and with _list = null at the end of DoWork -> memory increases

In cases where the comment _list = null commented out, it is not surprising to see this result. Because there is still a link to _list. Even if DoWork is never called again, the GC cannot know it.

In the third case, the garbage collector has the behavior that we expect from it.

In the fourth case, the BlockingCollection stores Data when you pass it as an argument to collection.Add(new Data(l)); but then what is done?

• A new Data structure has been created with data._list equal to l (i.e. since the List type is a class (reference type), data._list is equal to struct Data at the address of l ).
• Then you pass it as an argument to collection.Add(new Data(l)); , then it creates a copy of Data created in 1. Then the address l copied.
• The block block stores your Data elements in an array.
• When DoWork does _list = null , it removes the link to the problematic List only in the current structure, and not in all copied versions that are stored in the BlockingCollection .
• Then you have a problem if you do not clear the BlockingCollection .

How to find a problem?

To find a memory leak problem, I suggest you use SOS ( http://msdn.microsoft.com/en-us/library/bb190764.aspx ).

Here I will tell you how I found the problem. Since this is a problem that involves not only a heap, but also a stack, using heap analysis (as here) is not the best way to find the source of the problem.

1 Put a breakpoint on _list = null (because this line should work !!!)

2 Run the program

3 . When the breakpoint is reached, load the SOS debugging tool (write ".load sos" in the Immediate window)

4 The problem seems to come from private List> _list , which is saved correctly. Therefore, we will try to find instances of the type. Type !DumpHeap -stat -type List in the Immediate window. Result:

 total 0 objects Statistics: MT Count TotalSize Class Name 0570ffdc 1 24 System.Collections.Generic.List1[[System.Threading.CancellationTokenRegistration, mscorlib]] 04f63e50 1 24 System.Collections.Generic.List1[[System.Security.Policy.StrongName, mscorlib]] 00202800 2 48 System.Collections.Generic.List1[[System.Collections.Generic.Dictionary2[[System.String, mscorlib],[System.String, mscorlib]], mscorlib]] Total 4 objects 

The problematic type is the last List<Dictionary<...>> . There are 2 instances, and MethodTable (type reference type) is 00202800 .

5 To get the links, enter !DumpHeap -mt 00202800 . Result:

  Address MT Size 02618a9c 00202800 24 0733880c 00202800 24 total 0 objects Statistics: MT Count TotalSize Class Name 00202800 2 48 System.Collections.Generic.List1[[System.Collections.Generic.Dictionary2[[System.String, mscorlib],[System.String, mscorlib]], mscorlib]] Total 2 objects 

Two copies are shown with their addresses: 02618a9c and 0733880c

6 To find how they refer: type !GCRoot 02618a9c (for the first instance) or !GCRoot 0733880c (for the second). Result (I did not copy the entire result, but retained an important role):

 ESP:3bef9c:Root: 0261874c(ConsoleApplication1.Test1)-> 0261875c(System.Collections.Concurrent.BlockingCollection1[[ConsoleApplication1.Data, ConsoleApplication1]])-> 02618784(System.Collections.Concurrent.ConcurrentQueue1[[ConsoleApplication1.Data, ConsoleApplication1]])-> 02618798(System.Collections.Concurrent.ConcurrentQueue1+Segment[[ConsoleApplication1.Data, ConsoleApplication1]])-> 026187bc(ConsoleApplication1.Data[])-> 02618a9c(System.Collections.Generic.List1[[System.Collections.Generic.Dictionary2[[System.String, mscorlib],[System.String, mscorlib]], mscorlib]]) 

for the first instance and:

 Scan Thread 5216 OSTHread 1460 ESP:3bf0b0:Root: 0733880c(System.Collections.Generic.List1[[System.Collections.Generic.Dictionary2[[System.String, mscorlib],[System.String, mscorlib]], mscorlib]]) Scan Thread 4960 OSTHread 1360 Scan Thread 6044 OSTHread 179c 

for the second (when the analyzed object does not have a deeper root, I think this means that it has a link on the stack).

A look at 026187bc(ConsoleApplication1.Data[]) should be a good way to understand what will happen, because we finally see our Data type.

7 To display the contents of an object, use !DumpObj 026187bc or in this case, since it is an array, use !DumpArray -details 026187bc . Result (partial):

 Name: ConsoleApplication1.Data[] MethodTable: 00214f30 EEClass: 00214ea8 Size: 140(0x8c) bytes Array: Rank 1, Number of elements 32, Type VALUETYPE Element Methodtable: 00214670 [0] 026187c4 Name: ConsoleApplication1.Data MethodTable: 00214670 EEClass: 00211ac4 Size: 12(0xc) bytes File: D:\Development Projects\Centive Solutions\SVN\trunk\CentiveSolutions.Renderers\ConsoleApplication1\bin\Debug\ConsoleApplication1.exe Fields: MT Field Offset Type VT Attr Value Name 00202800 4000001 0 ...lib]], mscorlib]] 0 instance 02618a9c _list [1] 026187c8 Name: ConsoleApplication1.Data MethodTable: 00214670 EEClass: 00211ac4 Size: 12(0xc) bytes File: D:\Development Projects\Centive Solutions\SVN\trunk\CentiveSolutions.Renderers\ConsoleApplication1\bin\Debug\ConsoleApplication1.exe Fields: MT Field Offset Type VT Attr Value Name 00202800 4000001 0 ...lib]], mscorlib]] 0 instance 6d50950800000000 _list [2] 026187cc Name: ConsoleApplication1.Data MethodTable: 00214670 EEClass: 00211ac4 Size: 12(0xc) bytes File: D:\Development Projects\Centive Solutions\SVN\trunk\CentiveSolutions.Renderers\ConsoleApplication1\bin\Debug\ConsoleApplication1.exe Fields: MT Field Offset Type VT Attr Value Name 00202800 4000001 0 ...lib]], mscorlib]] 0 instance 6d50950800000000 _list 

Here we have the value of the _list attribute for the first three elements of the array: 02618a9c , 6d50950800000000 , 6d50950800000000 . I suspect 6d50950800000000 is a null pointer.

Here we have the answer to your question: there is an array (referred to by the lock collection (see 6.)) that directly contains the address _list , which we want the garbage collector to complete.

8 To ensure that it does not change when the _line = null line is executed, the line is executed.

Note

As I said before, using DumpHeap is not good for the current task, which implies value types. What for? Because value types are not on the heap, but on the stack. This is very easy to see: try it !DumpHeap -stat -type ConsoleApplication1.Data at the breakpoint. Result:

 total 0 objects Statistics: MT Count TotalSize Class Name 00214c00 1 20 System.Collections.Concurrent.ConcurrentQueue`1[[ConsoleApplication1.Data, ConsoleApplication1]] 00214e24 1 36 System.Collections.Concurrent.ConcurrentQueue`1+Segment[[ConsoleApplication1.Data, ConsoleApplication1]] 00214920 1 40 System.Collections.Concurrent.BlockingCollection`1[[ConsoleApplication1.Data, ConsoleApplication1]] 00214f30 1 140 ConsoleApplication1.Data[] Total 4 objects 

There is an array of Data , but not Data . Because DumpHeap parses only a bunch. Then !DumpArray -details 026187bc pointer is still here with the same value. And if you compare the roots of two instances that we discovered earlier (with !GCRoot ), only the line will be deleted before and after the line is executed. Indeed, list binding is only removed from 1 copy of the Data value type.