Geek Answers Handbook

Improve performance in IO file in C

I need to write a mass of integers to a file after performing heap operations on them one by one. I am trying to merge sorted files into one file. At the moment, I am writing a file after each operation. I use a mini-heap to merge files.

My questions -

  • When a file is being written, the disk is accessed each time a file is written or blocks of memory blocks are written

  • Will performance improve if I select heap output in an array of 1024 or more, and then record right away?

Thanks in advance.

EDIT - Will setbuffer () be used? I believe this should help to some extent.

0
source share
3 answers
1. When performing file write, is disk accessed every time a file write is made or chunks of memory blocks are written at a time?

Not. Your output is not written until the output buffer is full. You can force write with fflush to clear output streams, causing an immediate write, but otherwise the output is buffered.

 other 1. Will it improve performance if I'll take output of heap in an array of say size 1024 or may be more and then perform a write at once?

If you do not exhaust the heap, then no, you will not get significant performance by pushing the storage on the stack, etc. Buffering is always preferable, but if you save all the data in an array and then call write, you still have an output buffer of the same size you are dealing with.

+1
source

When a file is being written, is the disk accessed each time a file is written, or are pieces of memory blocks recorded at the same time?

It depends on the kernel. Buffers are cleared when fsync() called in the file descriptor. fflush() only clears data buffered in the FILE structure, does not flush kernel buffers.

Will performance be improved if I select the output of the heap in an array, say, size 1024 or maybe more, and then record right away?

I ran tests some time ago to compare write() and fwrite() performance with a custom implementation, and it turns out you can get fair acceleration by calling write() directly with large chunks. This is actually what fwrite() does, but because of the infrastructure it needs to support, it is slower than a regular implementation. As for the buffer size, 1024 is, of course, too small. 8K or something will work better.

+1
source

This is a specific operating system and implementation.

On most Linux systems — with a good file system such as Ext4 — the kernel will try to avoid disk access by caching a lot of file system data. See linuxatemyram

But I would still recommend avoiding too many I / O operations and have some buffering (when using stdio (3) routines, skip buffers from a few tens of kilobytes to fwrite (3) and use setvbuf (3) and fflush (3) with caution, and also use direct system calls, such as write (2) or mmap (2) with buffers, for example, 64 Kbytes ...)

BTW, using possibly posix_fadvise (2) syscall, can help performance a bit (if you use them wisely).

In reality, the bottleneck is often hardware. Use RAM file systems ( tmpfs ) or fast SSDs if you can.

On Windows systems (which I never used) I have no idea, but the general intuition is that some buffering should help.

0
source

More articles:


All Articles