In C ++, what is the fastest way to load a large binary file (1GB-4GB) into memory?

Question

On a 64-bit version of Linux (like an Amazon EC2 instance) I need to load a couple of large binaries into memory. What is the fastest way?

• ifstream
• Fread
• Posix open
• POSIX mmap (does not actually load the entire file into memory, which degrades performance)
• something else?

In addition, node may or may not run this executable file a second time, so it helps if the file loads even faster on subsequent attempts. Maybe some kind of preload step.

Answer 1

Time will depend on disk I / O, so the API you use is not as important as thinking about how the disk works. If you accidentally press a disk (rotating media), it will cost from 3 to 9 milliseconds to search ... after the disk is broadcast, it can withstand about 128 MB / s, that is, how quickly bits are discharged from the disk head . The SATA bus or PCIe bus has much higher bandwidth (600-2000 MB / s). Linux has a page cache in memory, where it stores a copy of the pages on disk, so assuming your computer has enough consecutive RAM attempts will be fast even if you accidentally access data. Therefore, advice is read in large blocks at a time. If you really want to speed up the boot, you can use mmap to map the entire file (1GB-4GB) and have an auxiliary stream that reads the 1st byte of each page in order.

Read more about disk drive specifications here.

Here you can find out more about the page cache .

Answer 2

Given the above information, I would say that mmap is a good candidate. Here are a few reasons why I say: 1. It gives you a WHOLE file without actually downloading (any) file until this part is needed. This is an advantage for fast loading, but if you end up going through each byte (or touch each section of the 4KB file), then there is not much difference. 2. mmap will only copy ONCE data from disk to your pages. This is more efficient in my testing than reading using fread or read on Linux (note also that the difference between fread and read for large reads can be ignored. In functions in C. C ++ streams, to add a fair bit of overhead costs, however, in my experience [I have tried many times various forms of this time].

As always, benchmarking always trumps the Internet. So you CAN FIND that what I said above is not appropriate for your circumstances. And, as indicated, if the code is good enough, any overhead in the code overshadows the speed at which drives can deliver data, even if you have a very fantastic RAID system with lots of parallel (SSD?) Drives, etc., in the end As a result, the transfer speed of the disk will be where the bottleneck is. All you can do at this moment is to try as little as possible other overhead costs, and get the data into the application as quickly as possible after the disk has delivered the data.

A good reference for bytes per second is to use dd if=/dev/zero of=somefile bs=4K count=1M (which writes the file, then you might want dd if=somefile of=/dev/null bs=4K to see how well you can read from disk.

Answer 3

You can try mmap with the flag MAP_POPULATE . I doubt you can do it faster.