Mmap to write sequential log file for speed?

Question

I want to write a log file, unstructured format (one line at a time) using mmap(for speed). What is the best procedure? Open a blank file, truncateup to 1 page size (write a blank line to resize the file?), Then mmap- and repeat if the filled area is mmaped?

I usually use mmapto record structures of a fixed size, as a rule, only one page at a time, but this is for writing log files (anywhere from 0.5 to 10 GB) using mmap, but I'm not sure what is best after the first mmaped area is filled - munmap, resize the file truncateand mmapthe next page?

When writing logs to a memory area, I would track the size and msyncwhat is the correct processing when I get to the end of the displayed memory area?

Let's say I never need to return or overwrite existing data, so I only write new data to a file.

Q1: When I get to the end of the displayed area, I make the file munmap, ftruncateto change the size of another page and mmapthe next page?

Q2: Is there a standard way of pre-emptive verification and is the next page in memory ready for the next record? Do it in a different thread when we get closer to the end of the displayed area?

Q3: Am I madvisefor serial access?

This is for real-time data processing with the requirement to store a log file - currently I'm just writing to a file. The log file is unstructured, text format, line-based.

This is for Linux / C ++ / c optional testing on Mac (so do not reassign [?]).

Any links / pointers to best practices are appreciated.

Answer 1

Fwrite VS mmap ( " " ). , , . fwrite mmap. mmap ; , fwrite, .

---

remapSize - , . fileSize , mappedSpace mmap ( ), alreadyWrittenBytes - , .

:

void init() {
  fileDescriptor = open(outputPath, O_RDWR | O_CREAT | O_TRUNC, (mode_t) 0600); // Open file
  result = ftruncate(fileDescriptor, remapSize); // Init size
  fsync(fileDescriptor); // Flush
  memoryMappedFile = (char*) mmap64(0, remapSize, PROT_WRITE, MAP_SHARED, fileDescriptor, 0); // Create mmap
  fileSize = remapSize; // Store mapped size
  mappedSpace = remapSize; // Store mapped size
}

Q1:

"Unmap-Remap" -.

• (msync)
• .

:

void unmap() {
  msync(memoryMappedFile, mappedSpace, MS_SYNC); // Flush
  munmap(memoryMappedFile, mappedSpace)
}

Remap , .

Remap

:

void fullRemap() {
  ftruncate(fileDescriptor, mappedSpace + remapSize); // Make file bigger
  fsync(fileDescriptor); // Flush file
  memoryMappedFile = (char*) mmap64(0, mappedSpace + remapSize, PROT_WRITE, MAP_SHARED, fileDescriptor, 0); // Create new mapping on the bigger file
  fileSize += reampSize;
  mappedSpace += remapSize; // Set mappedSpace to new size
}

:

void smallRemap() {
  ftruncate(fileDescriptor, fileSize + remapSize); // Make file bigger
  fsync(fileDescriptor); // Flush file
  remapAt = alreadyWrittenBytes % pageSize == 0 
            ? alreadyWrittenBytes 
            : alreadyWrittenBytes - (alreadyWrittenBytes % pageSize); // Adjust remap location to pagesize
  memoryMappedFile = (char*) mmap64(0, fileSize + remapSize - remapAt, PROT_WRITE, MAP_SHARED, fileDescriptor, remapAt); // Create memory-map
  fileSize += remapSize;
  mappedSpace = fileSize - remapAt;
}

mremap function,

Linux         .

Q2:

, . " ", , ( , ). . Q3 , .

Q3:

madvise MADV_SEQUENTIAL, , ​​ , .

Excerp man:

​​

:

mmap . "" , , fwrite.

mmap .

, . , mmap , .

Answer 2

mmap ( ). ?

mmap, write. . , mmap - ?

mmap , . ,

• ( , )
• ( , )
• mmap , . , , .

mmap, , .

:

http://lkml.iu.edu/hypermail/linux/kernel/0004.0/0728.html

[e-mail] > -  : >

, , mmap/mlock    3 , .    mmap/mlock , .   .

mmap() , .

, . , , , Improvment.

mmap:

• . . , . . TLB, .

• . , .

mmap:

• ( ), , - , mmap() - , . , ( - ), , , mmap() . mmap(), , .

• , mmap() - , . , , .

, , .

( ), , pessimal mmap().