Mmap is slower than getline?

Question

I am facing the problem of reading / writing files (in Gigs) line by line.

Reading many posts and forum sites (including the SO group), mmap was suggested as the fastest option for reading / writing files. However, when I implement my code using the readline and mmap methods, mmap is the slower of the two. This is true for both reading and writing. I tested files of ~ 600 MB in size.

My implementations are parsed line by line and then tokenized string. I will only provide file input.

Here is the getline implementation:

void two(char* path) { std::ios::sync_with_stdio(false); ifstream pFile(path); string mystring; if (pFile.is_open()) { while (getline(pFile,mystring)) { // c style tokenizing } } else perror("error opening file"); pFile.close(); } 

and here is mmap :

 void four(char* path) { int fd; char *map; char *FILEPATH = path; unsigned long FILESIZE; // find file size FILE* fp = fopen(FILEPATH, "r"); fseek(fp, 0, SEEK_END); FILESIZE = ftell(fp); fseek(fp, 0, SEEK_SET); fclose(fp); fd = open(FILEPATH, O_RDONLY); map = (char *) mmap(0, FILESIZE, PROT_READ, MAP_SHARED, fd, 0); /* Read the file char-by-char from the mmap */ char c; stringstream ss; for (long i = 0; i <= FILESIZE; ++i) { c = map[i]; if (c != '\n') { ss << c; } else { // c style tokenizing ss.str(""); } } if (munmap(map, FILESIZE) == -1) perror("Error un-mmapping the file"); close(fd); } 

I missed a lot of error checking for brevity.

Is my mmap implementation incorrect and thus affects performance? Maybe mmap is not ideal for my application?

Thanks for any comments or help!

Answer 1

The real power of mmap can freely search the file, use its contents directly as a pointer, and avoid the overhead of copying data from the kernel cache to user space. However, your sample code does not use this.

In your loop, you scan the buffer one character at a time, adding to stringstream . stringstream does not know how long the string takes, and so you have to redistribute it several times in the process. At this point, you lost any performance boost with mmap - even the standard getline implementation avoids a lot of redistributions (using the 128-byte buffer on the stack in the GNU C ++ implementation).

If you want to use mmap for maximum power:

• Do not copy your lines. Generally. Instead, copy the pointers directly to the mmap buffer.
• Use built-in functions like strnchr or memchr to find newlines; they use manual assembler and other optimizations to run faster than most open source search cycles.

Answer 2

The one who told you to use mmap is not very versed in modern machines.

The benefits of mmap performance is a complete myth. In the words of Linus Torvalds :

Yes, the memory is "slow", but hell, too, mmap ().

The problem with mmap is that every time you first touch a page in the displayed area, it gets into the kernel and actually displays the page in your address space, playing chaos with TLB.

Try a simple benchmark that reads a large 8K file at a time using read and then again with mmap . (Using the same 8K buffer over and over.) You will almost certainly find that read is actually faster.

Your problem has never been to get data from the kernel; it was with how you process the data after that. Minimize the work you do in different ways; just scan to find a new line, and then do one operation on the block. Personally, I will return to the implementation of read using (and reusing) a buffer that fits into the L1 cache (about 8 KB).

Or at least I would try a simple read vs. test mmap to find out what's happening on your platform faster.

[Update]

I found a couple more comments from Mr. Torvalds:

http://lkml.iu.edu/hypermail/linux/kernel/0004.0/0728.html http://lkml.iu.edu/hypermail/linux/kernel/0004.0/0775.html

Summary:

And besides, you still have the cost of missing CPU TLB, etc. Which can often be avoided if you simply re-read in the same area instead of being overly smart with memory management just to avoid copying.

memcpy () (ie, "read ()" in this case) will always be faster in many cases, only because it avoids all the extra complexity. While mmap () will be faster in other cases.

In my experience, reading and processing a large file sequentially is one of the “many cases” where using (and reusing) a moderate read / write buffer is much better than mmap .

Answer 3

You can use memchr to search for line endings. This will be much faster than adding one character at a time to a stringstream .

Answer 4

You use stringstream to store the strings you identify. This is not comparable with the implementation of getline, the stream of lines itself adds additional overhead. Like others, you can save the beginning of a line as char* and possibly the length of the line (or a pointer to the end of the line). The body of the read will be something like this:

 char* str_start = map; char* str_end; for (long i = 0; i <= FILESIZE; ++i) { if (map[i] == '\n') { str_end = map + i; { // C style tokenizing of the string str_start to str_end // If you want, you can build a std::string like: // std::string line(str_start,str_end); // but note that this implies a memory copy. } str_start = map + i + 1; } } 

Please also note that this is much more efficient because you do not process anything in each char (in your version you added a character to stringstream ).