The fastest way to check if memory is reset

Question

I have a program that should check if a fragment of a file is nullified or contains data. This alg works for an entire file up to several gigs in size and takes time. Is there a better way to check if it is reset to zero?

Platform: Linux and windows

bool WGTController::isBlockCompleted(wgBlock* block) { if (!block) return false; uint32 bufSize = (uint32)block->size; uint64 fileSize = UTIL::FS::UTIL_getFileSize(m_szFile); if (fileSize < (block->size + block->fileOffset)) return false; char* buffer = new char[bufSize]; FHANDLE fh=NULL; try { fh = UTIL::FS::UTIL_openFile(m_szFile, UTIL::FS::FILE_READ); UTIL::FS::UTIL_seekFile(fh, block->fileOffset); UTIL::FS::UTIL_readFile(fh, buffer, bufSize); UTIL::FS::UTIL_closeFile(fh); } catch (gcException &) { SAFE_DELETEA(buffer); UTIL::FS::UTIL_closeFile(fh); return false; } bool res = false; for (uint32 x=0; x<bufSize; x++) { if (buffer[x] != 0) { res = true; break; } } SAFE_DELETEA(buffer); return res; } 

Answer 1

How long is “some time”? ... I would say, trying to compare as many values ​​as possible in parallel, perhaps use some SIMD instructions to compare more than 4 bytes at a time?

Remember, however, that no matter how fast you compare, in the end, the data should still be read from the file. If the file is not already in the cache somewhere in memory, you can limit it to a size of 100-150 MB / s maximum before the disk bandwidth is saturated. If you have already reached this point, you may need to first look at an approach that avoids downloading a file or simply accepting the fact that it will not be faster.

Answer 2

Are there places in the file / fragment where non-zero values ​​are more likely? You need to find only one nonzero value (an interruption condition), so first find the places where you are likely to find them - which should not be the beginning of the file / piece. It may make sense to start at the end or check 1/3 in the middle, depending on the actual application.

However, I would not recommend accidentally moving to different positions; reading from disk can be incredible;) ..

Answer 3

I would like to see the build output for this function. Something you could do would speed it up to use SSE instructions. Using these instructions, you can download 8 bytes at a time, check them all for zero, and continue. And you can expand this for the loop several times.

Answer 4

You seem good, but you can heuristically optimize the starting position if you know in advance what type of file you will get ... but again, if this is a specific file, most likely the information will be in the header (first few bytes).

Also make sure the block-> size is not equal to 1 from whoever calls the method :)

Also check out the features of Boost memory mapped files ... Perhaps this will help, depending on how you calculate the optimal block size -> size

Answer 5

I have the answer “out of the box” for you, but I’m not sure how much it is possible to implement in your situation.

If you do not control the dumping process: since this is a large recovery file (dump?) Created in an excpetional case, why not scan the file (for 0 bytes) with a low priority immediately after resetting it and mark it somehow for faster identification later? (or you can pin it and parse / scan the zip file later)

Or, if you control the dumping process: (a slow process, which you should do anyway), why not indicate at the end of the dump file (or go back and write at the beginning of it) if the dump file is filled with 0 or has some reliable data (since you wrote this and know what is in it)? so you don’t have to pay twice for I / O.

The goal is to make reading much faster by defragmenting procss at another time in the hierarchy, because when a dump occurs, the operator is unlikely to wait for it to load.

Answer 6

First, do not allocate a new buffer each time. Select one (per stream) and reuse it. Use a nice big chunk and do some reading / checking passes.
Secondly, do not compare each character. Make comparisons with a larger integral type. Most likely you will need a 32-bit int, but depending on your os / compiler it might be faster to use a 64 or even a 128-bit int. With a 32-bit int, you reduce the number of comparisons by 4 times. Of course, your will must worry about the end conditions. For this, it is easy if the buffer you are comparing is not even a multiple of your int size, just set the last X bytes to 0 before doing the comparison. Thirdly, it can help your compiler unroll a little loop. Make 4 or 8 comparisons in the body of the loop. This should help the compiler optimize the bit, as well as reduce the number of comparisons to exit the loop. Make sure your buffer is a multiple of your comparison type x the number of comparisons in the loop. Fourth, it may be faster to use (* pBuffer ++) instead of the buffer [i], especially if the buffer is large.

For any of them, of course, you will want to get some indicators and see what actually helps.

Answer 7

I’ll tell you about a dirty, intolerable and complicated way, but it can be more effective ... If you are dealing with sparse files, you are really bored and want to mess with the internal filesystem files, reuse, you can try to add a new function that returns a bitmap that indicates which blocks are mapped and which are not (those that are not displayed are nullified, for the rest you still have to check manually).

Yes, I know he's crazy and no one would ever want to do something like this xD