MemoryMappedFile with very slow CreateViewStream

Question

I am using a memory mapped file which is approx. 100 GB of data. When I call CreateViewStream on this file, it takes 30 minutes to create it, and it seems that due to the size of the memory mapped file, but why is it taking so long? Will it copy the entire file to managed memory?

It will take much longer when I write a file with a file stream and access it without rebooting. (Surprisingly)

Answer 1

I can not reproduce these problems. Here is the code I used for testing:

  static void Main(string[] args) { var sw = Stopwatch.StartNew(); var mmf = MemoryMappedFile.CreateFromFile(@"f:\test.bin"); var stream = mmf.CreateViewStream(); for (int i = 0; i < 100000; i++) { stream.ReadByte(); } Console.WriteLine(sw.Elapsed); } 

f:\test.bin is the 100 kilogram filled file that I generated for the purpose of this test. I can create a MemoryMappedFile and then run CreateViewStream and read 100,000 bytes from it in 3.7s.

Please provide a sample code that describes the behavior that you described, and I will be happy to separate it and see what happens.

Answer 2

It is difficult to answer without code, knowledge of your main memory and architecture. Therefore, I can only guess about some important pointers:

• Do you have enough RAM? . If you refer to an address that has not yet been loaded into RAM, a page error occurs behind the scenes and reads the data in RAM for you. Your program does not notice this activity because your thread is paused while a page error is being processed. Good article here .
• Another important point from the same article - you do not have control over how much MMF is stored in memory or for how long. This means that using MMF can push other things out of RAM, such as pages of code or data, that you will need soon. This leads to slower execution. I especially want to point out to anyone reading this answer to another answer here , so we have a clear idea of ​​how slow this slowness is in terms of CPU cycles.
• Then you create a stream. Streams are good for sequential access, while you may be trying to read / write to it randomly .

As for the end-to-end execution time of your code in the FileStream and MMF approach, I think you should run the tests again, because starting your first approach can lead to a cache warming up for the second. Then the results will not be correct.

As per MSDN documentation for MMF ,

Files with memory mapping allow programmers to work with extremely large files, since memory can be managed simultaneously, and they allow full, random access to the file without the need for searching.

The way MMF works is that the entire (or part) of the file is displayed as virtual memory, which is transparently unloaded and exited from memory by the operating system when accessing parts of the file. That is why MMFs are great for working with large files.

You can be smarter and read part of the entire file and perform random access using:

 using (var accessor = mmf.CreateViewAccessor(offset, length)) { //Here you have access to a specific part of the file } 

so that you have access to the view with the specified offset and size of your memory mapping of the mammoth file.