How does a large array allocate memory?

Question

I am looking for a way to keep a large 3d sparse array structure in memory without spending a lot of memory. Here I conducted an experiment with longs arrays:

using System; using System.Diagnostics; using System.Runtime; namespace ConsoleApp4 { public class Program { static Process proc = Process.GetCurrentProcess(); const int MB = 1024 * 1024; const int IMAX = 5; const int JMAX = 100000000; public static void ShowTextWithMemAlloc(string text) { proc.Refresh(); Console.WriteLine($"{text,-30}WS64:{proc.WorkingSet64/MB,5}MB PMS64:{proc.PrivateMemorySize64/MB,5}MB"); Console.ReadKey(); } public static void Main(string[] args) { Console.Write(" "); ShowTextWithMemAlloc("Start."); long[] lArray = new long[IMAX * JMAX]; long[] l1Array = new long[IMAX * JMAX]; long[] l2Array = new long[IMAX * JMAX]; long[] l3Array = new long[IMAX * JMAX]; ShowTextWithMemAlloc("Arrays created."); lArray[IMAX * JMAX - 1] = 5000; l1Array[IMAX * JMAX - 1] = 5000; l2Array[IMAX * JMAX - 1] = 5000; l3Array[IMAX * JMAX - 1] = 5000; ShowTextWithMemAlloc("Last elements accessed."); for (var i=IMAX-1; i>= 0; i--) { for (var j=0; j<JMAX; j++) { lArray[i * JMAX + j] = i * JMAX + j; } ShowTextWithMemAlloc($"Value for row {i} assigned."); } //lArray = new long[5]; //l1Array = null; //l2Array = null; //l3Array = null; //GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce; //GC.Collect(); //ShowTextWithMemAlloc($"GC.Collect done."); ShowTextWithMemAlloc("Stop."); } } } 

If you want to test it, set the environment variable COMPlus_gcAllowVeryLargeObjects (Project Properties → Debugging) to 1 or change the JMAX. And this is the result:

  Start. WS64: 14MB PMS64: 8MB Arrays created. WS64: 15MB PMS64:15360MB Last elements accessed. WS64: 15MB PMS64:15360MB Value for row 4 assigned. WS64: 779MB PMS64:15360MB Value for row 3 assigned. WS64: 1542MB PMS64:15360MB Value for row 2 assigned. WS64: 2305MB PMS64:15361MB Value for row 1 assigned. WS64: 3069MB PMS64:15361MB Value for row 0 assigned. WS64: 3832MB PMS64:15362MB Stop. WS64: 3844MB PMS64:15325MB 

When I see that the memory consumption in the task manager looks like this in Process.WorkingSet64. What is the real number? Why is memory allocated when assigned? Is the array actually contiguous allocated memory? Is an array an array? Are there aliens? (dramatic background music)

Episode 2: We make a small change:

  //lArray[i * JMAX + j] = i * JMAX + j; var x= lArray[i * JMAX + j]; 

and nothing changes (at the output). Where is the difference between existing and non-existent? (more dramatic background music) Now we are waiting for a response from one of the mysterious people (they have a certain number and a small "k" under their names).

Episode 3: Another change:

  //lArray[IMAX * JMAX - 1] = 5000; //l1Array[IMAX * JMAX - 1] = 5000; //l2Array[IMAX * JMAX - 1] = 5000; //l3Array[IMAX * JMAX - 1] = 5000; //ShowTextWithMemAlloc("Last elements accessed."); long newIMAX = IMAX-3; long newJMAX = JMAX / 10; for (var i=0; i<newIMAX; i++) { for (var j=0; j<newJMAX; j++) { lArray[i * newJMAX + j] = i * newJMAX + j; //var x= lArray[i * JMAX + j]; } //ShowTextWithMemAlloc($"Value for row {i} assigned."); } ShowTextWithMemAlloc($"{newIMAX*newJMAX} values assigned."); 

Exit:

  Start. WS64: 14MB PMS64: 8MB Arrays created. WS64: 15MB PMS64:15369MB 20000000 values assigned. WS64: 168MB PMS64:15369MB Stop. WS64: 168MB PMS64:15369MB 

PMS64 for a single array (15369-8) / 4 = 3840 MB This is not a sparse array, but a partially filled array;). I use the full one is 168 MB.

The answer to some question: "Why aren't you using the exact size?" Because I don't know that? Data can come from several custom SQL queries. "Why don't you resize it?" Resize, create a new array and copy the values. This is the time to copy, memory, and at the end an evil GC comes and eats you.

I have lost my memory. (I don’t remember aliens ?!) And when yes, how much? 0, (3840-168) MB or (15369-8-168) MB?

Epilogue:

Is a comment a comment or an answer?

- continuous continuous memory?

Do the answers answer? Mysterious. ( more music )

( Scully : Mulder, the toads just fell from the sky! Mulder : I think their parachutes didn't open.)

Thanks everyone!

Answer 1

The working set is not the amount of allocated memory. This is the set of pages that are currently available for the process. Windows implements various policies around this, and the number is usually difficult to interpret.

Here, most likely, the requested memory was reset to zero from the OS. The first access to the page actually makes the page with a null value accessible.

You should look at private bytes.

You cannot allocate .NET arrays. You should probably look at using some kind of data structure that gives the impression of a sparse array.

Is the array actually contiguous allocated memory?

Yes, in terms of CLR and .NET code. The OS can play tricks, for example, with a lazy crash on the pages during the first reading or writing.

In "Episode 2," the answer is that an error occurs both for reading and writing. I don’t quite understand what episode 3 does, but I guess it just affects fewer pages.

I lost my memory

This is harder to say. As long as the pages are not affected, they are not physically used. They can be used for file cache, for example, or for other resident workgroup jobs. However, they do count on fixing the system. Windows ensures that it can make these pages available to you. You will not run out of memory with some random access to memory. Linux does not guarantee this. He has an OOM killer as a mitigation.

In extreme cases, if you allocate 1 TB, you need to have more than 1 TB of RAM and the size of the swap file, even if none of these places can be used.

Consider the use of memory mapped files. Here the file is a backup storage, and the RAM is considered as a cache. It will behave exactly the same.