Which one is faster, reading from disk or allocating system memory

Question

My XP environment is 32-bit. I find that when the allocated memory has a maximum size of 2 GB, this means that a bit of virtual space is available, and the allocation of memory is very slow.

So, if I have a sample file, my application should analyze them. I have two ways. One of them is to read them all in the system memory, and then conduct an analysis. Another is to first save the memory buffer as a cache and read part of the page file in this buffer, analyze and then discard it, then read the second part of the page file and redefine the cache, repeat the analysis.

Of profiling, it looks second faster since it avoids the time spent on posting.

What do you think? Thanks in adavance.

Answer 1

(1) I'm not sure the question matches the name. If you place about 2 GB of RAM on 32-bit Windows, the system will probably map a lot of memory to disk and that I will first look for slowdowns. When you use a lot of memory, you should think of it as storage on disk (in the pagefile.sys file) but cached in physical memory. The second one may be faster not because of the cost of allocation, but because of the cost of using a large amount of memory at the same time. In fact, when you copy a file to one large selection, you copy most of the disk disk-> disk through RAM, and then when you run it again for analysis, you load the copy back into RAM. If your analysis is a one-pass algorithm in which there is a lot of redundant work.

(2) I think the mmap file (MapViewOfFile and friends on Windows).

Edit: (3) caution. If this file is currently 1.8 GB, it might be 4 GB next year. If so, I plan now to have a size larger than 2 ^ 32 on a 32-bit machine, which means either accepting the second option, or still using MapViewOfFile, but not executing it at one reasonable file size. Otherwise, you will review this code the first time you run it in a large file and report an error.

Answer 2

You forgot the 3d path - to map memory to a file, see the CreateFileMapping / MapViewOfFile function. This is the fastest way.

Answer 3

It is best to use MapViewOfFile windows and similar functions (Windows equivalent for mmap). This will allow the operating system to manage paging in various parts of the file.

Answer 4

Why is the amount of allocated memory so high? If memory allocations take up enough time, then you will find that doing it in memory is much faster - my approach would be to do this in memory, and try to find a way to reduce memory usage to such an extent.

Answer 5

As I see the situation, you either control the paging yourself, or let the operating system manage the search call for you. In most cases, I suggest allowing the operating system to handle paging (use virtual memory). Since I have a distrust of MS operating systems, I cannot recommend this technique, although your mileage may vary.