Automatic selection of buffer size for file input / output

Question

I have some pretty serious doubts. Often I have to write applications that use buffered file I / O, and every time I encounter the problem of selecting the buffer size, I end up making a trial and error often with pretty unpleasant results. I want to know if there is any method or algorithm that can automatically determine the optimal buffer size for a job based on a base platform, such as Teracopy, when processing files on Windows. I mainly use Qt for a graphical interface.

If possible, a tiny example in C / C ++ / C # / Java is really appreciated!

Thanks!

Answer 1

In Java, the optimal size is usually equal to the size of the L1 cache, which is usually 32 KB. In Java, at least the choice of 1024 bytes or 1 MB doesn't really matter (<20%)

If you read data sequentially, usually your OS is smart enough to detect it and pre-program the data for you.

What you can do is the following. This test shows a significant difference in the used block sizes.

public static void main(String... args) throws IOException { for (int i = 512; i <= 2 * 1024 * 1024; i *= 2) readWrite(i); } private static void readWrite(int blockSize) throws IOException { ByteBuffer bb = ByteBuffer.allocateDirect(blockSize); long start = System.nanoTime(); FileChannel out = new FileOutputStream("deleteme.dat").getChannel(); for (int i = 0; i < (1024 << 20); i += blockSize) { bb.clear(); while (bb.remaining() > 0) if (out.write(bb) < 1) throw new AssertionError(); } out.close(); long mid = System.nanoTime(); FileChannel in = new FileInputStream("deleteme.dat").getChannel(); for (int i = 0; i < (1024 << 20); i += blockSize) { bb.clear(); while (bb.remaining() > 0) if (in.read(bb) < 1) throw new AssertionError(); } in.close(); long end = System.nanoTime(); System.out.printf("With %.1f KB block size write speed %.1f MB/s, read speed %.1f MB/s%n", blockSize / 1024.0, 1024 * 1e9 / (mid - start), 1024 * 1e9 / (end - mid)); } 

prints

 With 0.5 KB block size write speed 96.6 MB/s, read speed 169.7 MB/s With 1.0 KB block size write speed 154.2 MB/s, read speed 312.2 MB/s With 2.0 KB block size write speed 201.5 MB/s, read speed 438.7 MB/s With 4.0 KB block size write speed 288.0 MB/s, read speed 733.9 MB/s With 8.0 KB block size write speed 318.4 MB/s, read speed 711.8 MB/s With 16.0 KB block size write speed 540.6 MB/s, read speed 1263.7 MB/s With 32.0 KB block size write speed 726.0 MB/s, read speed 1370.9 MB/s With 64.0 KB block size write speed 801.8 MB/s, read speed 1536.5 MB/s With 128.0 KB block size write speed 857.5 MB/s, read speed 1539.6 MB/s With 256.0 KB block size write speed 794.0 MB/s, read speed 1781.0 MB/s With 512.0 KB block size write speed 676.2 MB/s, read speed 1221.4 MB/s With 1024.0 KB block size write speed 886.3 MB/s, read speed 1501.5 MB/s With 2048.0 KB block size write speed 784.7 MB/s, read speed 1544.9 MB/s 

What this test does not show is that the hard drive only supports 60 MB / s and 40 MB / s. All you are testing is speed and exit from the cache. If that was your only priority, you would use a memory mapped file.

 int blockSize = 32 * 1024; ByteBuffer bb = ByteBuffer.allocateDirect(blockSize); FileChannel out = new FileOutputStream("deleteme.dat").getChannel(); for (int i = 0; i < (1024 << 20); i += blockSize) { bb.clear(); while (bb.remaining() > 0) if (out.write(bb) < 1) throw new AssertionError(); } out.close(); long start = System.nanoTime(); FileChannel in = new FileInputStream("deleteme.dat").getChannel(); MappedByteBuffer map = in.map(FileChannel.MapMode.READ_ONLY, 0, in.size()); in.close(); long end = System.nanoTime(); System.out.printf("Mapped file at a rate of %.1f MB/s%n", 1024 * 1e9 / (end - start)); 

prints

 Mapped file at a rate of 589885.5 MB/s 

This is so fast because it simply maps the data in the OS cache directly to the application memory (therefore, copying is not required)

Answer 2

I see this code in C:

 #include <sys/types.h> #include <sys/stat.h> #include <unistd.h> #include <stdio.h> int main() { struct stat fi; stat("/", &fi); printf("%d\n", fi.st_blksize); return 0; } 

It returns the optimal block size. You need to use it for this. I am using a stream source to destination with a block size of 16 * in order to have optimal performance. Because this test will show the best with a PC idling with some hardware / OS. But not a real case.