DbContext OutOfMemoryException

Question

I have a DbContext with a dataset of> 20M records that needs to be converted to a different data format. So I read the data in memory, perform some tasks, and then delete the DbContext. The code works fine, but after a while I get OutOfMemoryExceptions. I was able to narrow it down to the next code snippet where I get 2M records, then release them and retrieve them again. The first search works fine, the second is an exception.

// first call runs fine using (var dbContext = new CustomDbContext()) { var list = dbContext.Items.Take(2000000).ToArray(); foreach (var item in list) { // perform conversion tasks... item.Converted = true; } } // second call throws exception using (var dbContext = new CustomDbContext()) { var list = dbContext.Items.Take(2000000).ToArray(); foreach (var item in list) { // perform conversion tasks... item.Converted = true; } } 

Shouldn't the GC automatically free up all the memory allocated in the first use block, so the second block should work just as well as the first?

In my actual code, I don't get 2 million records at once, but something between 0 and 30K at each iteration. However, after about 15 minutes, my memory runs out, although all the objects should have been released.

Answer 1

After refactoring, the memory is freed. I don’t know why, but it works.

 private static void Debug() { var iteration = 0; while(true) { Console.WriteLine("Iteration {0}", iteration++); Convert(); } } private static void Convert() { using (var dbContext = new CustomDbContext(args[0])) { var list = dbContext.Items.Take(2000000).ToList(); foreach (var item in list) { item.Converted = true; } } } 

When I move the contents of Convert () into a while loop in Debug (), OutOfMemoryExceptions is called.

 private static void Debug() { var iteration = 0; while(true) { Console.WriteLine("Iteration {0}", iteration++); using (var dbContext = new CustomDbContext(args[0])) { // OutOfMemoryException in second iteration var list = dbContext.Items.Take(2000000).ToList(); foreach (var item in list) { item.Converted = true; } } } } 

Answer 2

I suspect you met LOH. Your objects are probably larger than threashold and they get there, so the GC doesn't help by default.

Try the following: https://www.simple-talk.com/dotnet/.net-framework/large-object-heap-compaction-should-you-use-it/

and see if your exception has disappeared.

i.e. add this between the first and second parts:

 GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce; GC.Collect(); 

Answer 3

IEnumerable has GetEnumerator (), so you can try this to avoid .ToArray () or .ToList (), which are not needed if you just want to read:

 // first call using (var dbContext = new CustomDbContext()) { foreach (var item in dbContext.Items.Take(2000000)) { // perform conversion tasks... item.Converted = true; } } // second call using (var dbContext = new CustomDbContext()) { foreach (var item in dbContext.Items.Take(2000000)) { // perform conversion tasks... item.Converted = true; } } 

Answer 4

Starting a GC will not help you; you must run each iteration in a different context. And delete your context.

 // ID is your primary key long startID = 0; while(true){ using(var db = new CustomDbContext()){ var slice = db.Items.Where(x=>x.ID > startID) .OrderBy(x=>x.ID) .Take(1000).ToList(); // stop if there is nothing to process if(!slice.Any()) break; foreach(var item in slice){ // your logic... item.Converted = true; } startID = slice.Last().ID; } } 

If you want to handle these things faster, an alternative approach would be to run slices in parallel ....

Alternative approach

I would recommend using dividing slices at 100x100, then I can process 100 slices of 100 elements in parallel.

You can always easily customize slicing to meet your speed needs.

 public IEnumerable<IEnumerable<T>> Slice(IEnumerable<T> src, int size){ while(src.Any()){ var s = src.Take(size); src = src.Skip(size); yield return s; } } long startID = 0; while(true){ using(var db = new CustomDbContext()){ var src = db.Items.Where(x=>x.ID > startID) .OrderBy(x=>x.ID) .Take(10000).Select(x=>x.ID).ToList(); // stop if there is nothing to process if(!src.Any()) break; Parallel.ForEach(src.Slice(100), slice => { using(var sdb = new CustomDbContext()){ foreach(var item in sdb.Items.Where(x=> slice.Contains(x.ID)){ item.Converted = true; } } } ); startID = src.Last(); } }