Understanding fwrite () behavior buffering

Question

I use the fwrite() function call to write data to a pipe in Linux.

Previously, fwrite() called for small pieces of data (on average 20 bytes) several times, and buffering remained fwrite() . strace in the process showed that 4096 bytes of data were written each time.

It turned out that this recording process was a bottleneck in my program. So I decided to write the data in my code to 64 KB blocks, and then write the whole block at a time using fwrite() . I used setvbuf() to set the FILE * pointer to "No Buffering".

The performance improvement was not as significant as I expected.

More importantly, strace output showed that the data is still being written 4096 bytes at a time. Can someone please explain this behavior to me? If I call fwrite() with 64 KB of data, why does it only write 4096 bytes at a time?

Is there an alternative to fwrite() to write data to a pipe using the FILE * pointer?

Answer 1

The 4096 ships with the Linux machines that underlie piping. These are two places. One of them is pipeline throughput. Capacity is a single system page in earlier versions of Linux, which is 4096 bytes on a 32-bit i386 machine. (On more modern versions of Linux, the capacity is 64 KB.)

Another place that you encounter this 4096 byte problem is in the given constant PIPE_BUF , the number of bytes that are guaranteed to be processed atomically. On Linux, this is 4096 bytes. What this limit means depends on whether the conveyor is set to block or non-block. Make a man -S7 pipe for all gory details.

If you are trying to exchange huge amounts of data at high speed, you may need to rethink your use of pipes. You are in a Linux box, so shared memory is an option. You can use channels to send relatively small amounts of data as a signaling mechanism.

Answer 2

If you want to change the buffering behavior, you must do this immediately after fopen (or before any I / O, for standard file descriptors stdin , stdout , stderr ). You also do not want to disable buffering and you are trying to control the buffer yourself; rather, point your 64K buffer to setvbuf so that it can be used properly.

If you really want to control buffering manually, do not use stdio ; use the lower level calls open , write and close .