How to get a copy to write to work in shared memory on Linux

Question

I tried to write a small application to familiarize myself with the concept of copy-on-write in user space. I read the answer from MSalters and realized that it would work only if I start with the mmap 'ed file to store my data. Since I need a file-based persistent, I tried to do the same with shared memory. First, I mmap 'ed and initialized shm fd, then I displayed the second copy using MAP_PRIVATE and read it again. However, a simple reading from it makes the kernel copy all of this, taking up significantly more time and consuming twice as much memory. Why does this not do COW?

Here is the program I came up with to illustrate the behavior:

 #include <stdio.h> #include <stdlib.h> #include <unistd.h> #include <sys/mman.h> #include <fcntl.h> #include <assert.h> static const size_t ARRAYSIZE = 1UL<<30; void init(int* A) { for (size_t i = 0; i < ARRAYSIZE; ++i) A[i] = i; } size_t agg(const int* A) { size_t sum = 0; for (size_t i = 0; i < ARRAYSIZE; ++i) sum += A[i]; return sum; } int main() { assert(sizeof(int) == 4); shm_unlink("/cowtest"); printf("ARRAYSIZE: %lu\n", ARRAYSIZE); int fd = shm_open("/cowtest", O_RDWR | O_CREAT | O_TRUNC, 0); if (fd == -1) { perror("Error allocating fd\n"); return 1; } if (ftruncate(fd, sizeof(int) * ARRAYSIZE) == -1) { perror("Error ftruncate\n"); return 1; } /* Open shm */ int* A= (int*)mmap(NULL, sizeof(int) * ARRAYSIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); if (A == (int*)-1) { perror("Error mapping A to memory\n"); return 1; } init(A); /* Create cow copy */ int* Acopy = (int*)mmap(NULL, sizeof(int) * ARRAYSIZE, PROT_READ, MAP_PRIVATE, fd, 0); if (Acopy == (int*)-1) { printf("Error mapping copy from file\n"); return 1; } /* Aggregate over A */ size_t sumA = agg(A); size_t expected = (ARRAYSIZE * (ARRAYSIZE - 1)) >> 1; assert(expected == sumA); /* Aggregate over Acopy */ size_t sumCopy = agg(Acopy); assert(expected == sumCopy); shm_unlink("/cowtest"); printf("Enter to exit\n"); getchar(); return 0; } 

I compiled it with g++ -O3 -mtune=native -march=native -o shm-min shm-min.cpp -lrt .

The created array contains 4 GB of integer values. Right before the program ends, however, 8 GB of shared memory is allocated, and in /proc/<pid>/smaps you can see that in fact it completely copied during the read-only operation. I have no idea why this is. Is this a kernel bug? Or am I missing something?

Thanks so much for any ideas. Lars

Edit Here is the relevant content /proc/<pid>/smaps on Ubuntu 14.04 (3.13.0-24):

 7f3b9b4ae000-7f3c9b4ae000 r--p 00000000 00:14 168154 /run/shm/cowtest (deleted) Size: 4194304 kB Rss: 4194304 kB Pss: 2097152 kB Shared_Clean: 0 kB Shared_Dirty: 4194304 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 4194304 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Locked: 0 kB VmFlags: rd mr mw me sd 7f3c9b4ae000-7f3d9b4ae000 rw-s 00000000 00:14 168154 /run/shm/cowtest (deleted) Size: 4194304 kB Rss: 4194304 kB Pss: 2097152 kB Shared_Clean: 0 kB Shared_Dirty: 4194304 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 4194304 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Locked: 0 kB VmFlags: rd wr sh mr mw me ms sd 

Answer 1

There was no copying. There is a hint in the smaps file:

 Size: 4194304 kB Rss: 4194304 kB Pss: 2097152 kB 

See how Pss is half the real size of the displayed area? This is because it is divided into two uses (Pss = proportional total size). That is, you have the same file twice displayed in different ranges of virtual memory, but the main physical pages are the same for both comparisons.

To determine the physical addresses of the respective pages, you can use the tool here . Save it as page-types.c , run make page-types , and then ./page-types -p <pid> -l -N . You will see that different virtual addresses (in the first column) are displayed on the same physical pages (in the second column).

If you add the PROT_WRITE permission PROT_WRITE for the second mapping and call init(Acopy) , you will see that Pss goes to 4 GB and the physical addresses of the corresponding pages no longer match.

TL; DR COW is working.