Reading a file over the network is slow due to extra reads

Question

I am reading a file, and I am either reading a data row (1600 consecutive reads of 17 bytes) or a data column (1600 reads of 17 bytes, separated by 1600 * 17 = 27,200 bytes). The file is located either on the local drive or on the remote drive. I read 10 times, so I expect in each case to read in 272,000 bytes of data.

On the local drive, I see what I expect. On the remote drive, while reading sequentially, I also see what I expect, but when reading the column, I see a ton of additional readings. They have a length of 32,768 bytes and do not seem to be used, but they do a scan of the amount of data from 272,000 bytes anywhere from 79 MB to 106 MB. Here is the result using Process Monitor:

  1: 39: 39.4624488 PM DiskSpeedTest.exe 89628 ReadFile \\ BCCDC01 \ BCC-raid3 \ SeisWareInc Temp Dir \ BPepers_Temp \ Projects \ PT_4 \ Horizons \ BaseName3D_1 \ RR_AP SUCCESS Offset: 9,390,069, Length: 17
 1: 39: 39.4624639 PM DiskSpeedTest.exe 89628 FASTIO_CHECK_IF_POSSIBLE \\ BCCDC01 \ BCC-raid3 \ SeisWareInc Temp Dir \ BPepers_Temp \ Projects \ PT_4 \ Horizons \ BaseName3D_1 \ RR_AP SUCCESS Operation: Read ...
 1: 39: 39.4624838 PM DiskSpeedTest.exe 89628 ReadFile \\ BCCDC01 \ BCC-raid3 \ SeisWareInc Temp Dir \ BPepers_Temp \ Projects \ PT_4 \ Horizons \ BaseName3D_1 \ RR_AP SUCCESS Offset: 9,388,032, Length: O. Flags: 32,768 cached, Paging I / O, Synchronous Paging I / O, Priority: Normal
 1: 39: 39.4633839 PM DiskSpeedTest.exe 89628 ReadFile \\ BCCDC01 \ BCC-raid3 \ SeisWareInc Temp Dir \ BPepers_Temp \ Projects \ PT_4 \ Horizons \ BaseName3D_1 \ RR_AP SUCCESS Offset: 9,417,269, Length: 17
 1: 39: 39.4634002 PM DiskSpeedTest.exe 89628 FASTIO_CHECK_IF_POSSIBLE \\ BCCDC01 \ BCC-raid3 \ SeisWareInc Temp Dir \ BPepers_Temp \ Projects \ PT_4 \ Horizons \ BaseName3D_1 \ RR_AP SUCCESS Operation: Read 17, 17,
 1: 39: 39.4634178 PM DiskSpeedTest.exe 89628 ReadFile \\ BCCDC01 \ BCC-raid3 \ SeisWareInc Temp Dir \ BPepers_Temp \ Projects \ PT_4 \ Horizons \ BaseName3D_1 \ RR_AP SUCCESS Offset: 9,444,469, Length: 17
 1: 39: 39.4634324 PM DiskSpeedTest.exe 89628 FASTIO_CHECK_IF_POSSIBLE \\ BCCDC01 \ BCC-raid3 \ SeisWareInc Temp Dir \ BPepers_Temp \ Projects \ PT_4 \ Horizons \ BaseName3D_1 \ RR_AP SUCCESS Operation: Read, ...
 1: 39: 39.4634529 PM DiskSpeedTest.exe 89628 ReadFile \\ BCCDC01 \ BCC-raid3 \ SeisWareInc Temp Dir \ BPepers_Temp \ Projects \ PT_4 \ Horizons \ BaseName3D_1 \ RR_AP SUCCESS Offset: 9,441,280, Length: 32,768, I / O: 7,768, Non cached, Paging I / O, Synchronous Paging I / O, Priority: Normal
 1: 39: 39.4642199 PM DiskSpeedTest.exe 89628 ReadFile \\ BCCDC01 \ BCC-raid3 \ SeisWareInc Temp Dir \ BPepers_Temp \ Projects \ PT_4 \ Horizons \ BaseName3D_1 \ RR_AP SUCCESS Offset: 9,471,669, Length: 17
 1: 39: 39.4642396 PM DiskSpeedTest.exe 89628 FASTIO_CHECK_IF_POSSIBLE \\ BCCDC01 \ BCC-raid3 \ SeisWareInc Temp Dir \ BPepers_Temp \ Projects \ PT_4 \ Horizons \ BaseName3D_1 \ RR_AP SUCCESS Operation: 17, 69 Offset, Read Off, 17, 6969
 1: 39: 39.4642582 PM DiskSpeedTest.exe 89628 ReadFile \\ BCCDC01 \ BCC-raid3 \ SeisWareInc Temp Dir \ BPepers_Temp \ Projects \ PT_4 \ Horizons \ BaseName3D_1 \ RR_AP SUCCESS Offset: 9,498,869, Length: 17
 1: 39: 39.4642764 PM DiskSpeedTest.exe 89628 FASTIO_CHECK_IF_POSSIBLE \\ BCCDC01 \ BCC-raid3 \ SeisWareInc Temp Dir \ BPepers_Temp \ Projects \ PT_4 \ Horizons \ BaseName3D_1 \ RR_AP SUCCESS Operation: Read, 18, 69, Set Off, Off, 9.869, 89, 69, Set Off, Off
 1: 39: 39.4642922 PM DiskSpeedTest.exe 89628 ReadFile \\ BCCDC01 \ BCC-raid3 \ SeisWareInc Temp Dir \ BPepers_Temp \ Projects \ PT_4 \ Horizons \ BaseName3D_1 \ RR_AP SUCCESS Offset: 9,498,624, Length: 32,768, I / O: 7,768, Non cached, Paging I / O, Synchronous Paging I / O, Priority: Normal

Pay attention to additional readings of 32,768 with I / O flags configured for non-caching, paging I / O, synchronous paging I / O, priority: normal. These extra reads are something that takes from 272KB to 106MB and causes slowness. They do not happen when reading from a local file, or if I read a line so that everything is consistent.

I tried setting FILE_FLAG_RANDOM_ACCESS, but it doesn't seem to help. Any ideas on what causes these extra readings and how to stop them?

Tests are performed on a 64-bit Vista system. I can provide the source code for the program to demonstrate the problem, as well as the console program that runs the tests.

Answer 1

You may have encountered problems blocking orders over smb. As a rule, when reading / saving a file over network windows, you can drag the entire file to the client and send the changes back. When you work with flat file bases or files, this can cause unnecessary reads through the smb share.

I'm not sure if there is a way to just pull the whole file, read the lines from that file in a local copy, and then discard the changes or not.

You will read several nightmares about oplocks and flat database files.

http://msdn.microsoft.com/en-us/library/aa365433%28VS.85%29.aspx

Not sure if this will solve your problem, but it may cause you to point in the right direction. Good luck

Answer 2

I found the answer to this question. Windows does read files through the page cache, so when I read 17 bytes, you first need to transfer the full page with 32 KB, and then you can copy the 17 bytes that I want from the page cache. Unpleasant performance result!

The same thing happens the first time that reading is done in a local file, since in this case it still loads the full page at a time into the page cache. But the second time, when I run the test locally, all the files are already in the page cache, so I do not see it. And if SuperFetch is turned on, and I did these tests for a while, Windows will start loading the file into the cache before I even run the test application, so again I don’t see the pages being executed being executed.

Thus, the operating system does a lot of things behind the scenes, making it difficult to get good performance testing!

Answer 3

I see this all the time, and it’s not under your control: the network does what it wants.

If you know that the file will be less than 1 MB, just pull it all into memory.

Answer 4

My guess is that the OS does this on its own to read before the file when you need data later. If this does not hurt you, then it does not matter.

Check out the caching section of the CreateFile API section.

You might like to try FILE_FLAG_NO_BUFFERING to see if it stops unnecessary reads. Be careful, using this flag may slow down your application. Usually you use this flag if you understand how to quickly transfer data from disk, and OS caching only interferes.

You can also get the same behavior as a network file with local files if you use the FILE_FLAG_SEQUENTIAL_SCAN flag. This flag tells the window cache manager what you will do and will try to get data for you ahead of schedule.

Answer 5

I think SMB always passes a block, not a small set of bytes.

Some block size matching information can be found here. http://support.microsoft.com/kb/q223140

So, you see a read to copy the corresponding block, followed by local reads (s) of 17 bytes in the block. (If you look at the template, there are several pairs of 17 bytes where two reads fall into the same block).

The fix obviously depends on the control you have over the application, as well as the size and structure of the database. (for example, if the database had one column for each file, then all reads would be sequential. If you used the database server, you would not use SMB, etc.)

If this is some kind of consolation, iTunes is horrified to use a network drive .