What is the ideal memory block size for copying?

Question

I am currently using 100 megabytes per block of memory to copy large files.

Is there a “good” amount that people usually use?

Edit

Thanks for all the great answers.

I am still new to these concepts, so I will try to understand many of the ones that were mentioned (e.g. writeback cache). I keep learning new things :)

Answer 1

A typical option is to block between 4096 and 32 KB. Using 100 MB is counterproductive. You take up RAM with a buffer that can be used much better as a file system writeback cache.

Copying files occurs very quickly, when the file is completely cached, the WriteFile () call is a simple copy of memory to memory. Then the cache manager lazily writes it to disk. But when there is no more space in the cache, the copy speed drops from the cliff when WriteFile () has to wait until the space becomes available. Now it goes to disk write speed.

Answer 2

I would recommend that you compare this figure and do not forget to include much smaller block sizes. In my own tests, I got quite conflicting results.

When reading and writing from a hard disk, all (power of two) blocks from 512 to 512 kB in size gave the same speed. Increasing the block size from 512 kB to 1 MB reduced the copy speed to about 60%. Increasing the block size increased the speed even more, but never returned to the speed of using small blocks.

When all the copied data was in the cache, the copy speed (much faster) improved with increasing block sizes, smoothed around blocks to 32 kB, and then suddenly decreased to about half the previous speed when moving from 256 kB to 512 KB blocks, never going back to previous speeds.

After this test, I deleted the read / write block sizes in several of my programs from 1 MB to 32 kB.

Answer 3

As a rule, the use of large blocks is of little use.

Suppose your operating system is super-scientific, and each read or write operation will corrupt the hard drive (in practice, you will often find that writes are queued and read with read-forward buffering, which reduces the advantage of using large buffers in your application code) .

Then each block costs you (say) 2x10ms for two requests (one for reading and one for writing), and a small dot increases the size of your block as soon as the time for actual reading and writing is significantly longer. Really fast HD can read and write at a speed of 150 MB / s, in which case 10 ms will correspond to 1.5 MB read / write, and you will get little to lock beyond 15 MB.

In practice (1) your search time is likely to be less, (2) your read and write bandwidth is likely to be longer, and (3) your OS and disk hardware are likely to cache and operate things for you ; you will probably see little or no use of blocks larger than 100 KB.

(You should probably compare the different locks and see what you get on your own system.)

Answer 4

I think it depends on the size of free memory that you have.

If you use 100 M blocks for copying on a computer that has, for example, 30 MB of empty memory, it will take longer to copy than using a smaller (20 M) block.

If your copy buffer is larger than the size of free memory, then due to the exchange of virtual memory, copying will be slower than expected.

Answer 5

This is a pretty big amount. Think that you don’t even start writing data before reading 100 MB, so the file system driver does not even have the ability to write any destination file while reading. A disc can write parts of a file that go under the heading when it reads the source file (for example, look up lifting ).

Answer 6

Given what a drive should look for when it changes tracks, can a block size, for example 63 x 512 = 32256, get optimal results?