List of memory overheads

Question

I have something like this:

List<object> res = new List<object>(); ... while (...) { byte[] val = //get new byte[4] res.Add(val); //if this is commented memory usage goes from 2Gb to 180Mb } return res; 

Now val always a byte [4], and there are about 37000000 elements. Therefore, I think res should be around 37 * 10 ^ 6 * 4 bytes = 148 MB, but the actual memory usage is about 2 GB. If I comment on res.Add(val); , then the memory usage will be somewhere around 100 MB.

So where is the memory going?

EDIT

I tried to use uint instead of byte [4], but the result is the same:

EDIT2

Well, using uint instead of byte [4] and List<uint>() instead of List<object> puts the memory at about 300 MB.

Answer 1

This is a common problem of allocating a large number of small objects: the overhead of objects that you can usually ignore comes into play.

37 * 10 ^ 6 * 4 bytes = 148 MB

Assuming an array of four bytes occupies four bytes in memory, it is incorrect. In addition to a four-byte payload, an array object must store the length of the array, synchronization block, and type pointer. This is 12 bytes of overhead on a 32-bit system or 24 bytes on a 64-bit system.

In addition to the individual overhead of array objects, you need to consider the overhead of the memory allocator, the overhead of aligning the memory, and the overhead of the garbage collector. All things taken together, it is unreasonable to see that the total used memory is increased to 2 GB.

One way to fix this is to go to the list of uint s, which occupy four bytes each. When you need to save four bytes, convert them to uint and save them in a list. When you need your bytes, convert uint to a temporary four-byte array. Use BitConverter to deal with conversion between uint and byte s arrays.