How can I poll a large number of files for changes?

Question

I would like to poll the file system for any modified, added, or deleted files or subdirectories. All changes must be quickly detected, but without pressure on the machine. OS - Windows> = Vista, the monitored part is the local directory.

As a rule, I would resort to FileSystemWatcher, but this led to problems with other programs that tried to look at the same place (mainly Windows Explorer). In addition, I heard that FSW is not very reliable even for local folders and with a large buffer.

The main problem is that the number of files and directories can be very large (guess 7 digits). Just checking all the files every second noticeably affected my machine.

My next idea was to check different parts of the whole tree per second, in order to reduce the overall effect and possibly add heuristics, for example, checking files that often change in faster sequence.

I am wondering if patterns exist for this kind of problem, or if anyone has experience in this situation.

Answer 1

We have implemented a similar function using C #. FileSystemWatcher was inefficient with large directory trees.

Our alternative, used FSNodes , the structure we created using the following Windows API calls:

  [StructLayout(LayoutKind.Sequential)] private struct FILETIME { public uint dwLowDateTime; public uint dwHighDateTime; }; [StructLayout(LayoutKind.Sequential, CharSet=CharSet.Unicode)] private struct WIN32_FIND_DATA { public FileAttributes dwFileAttributes; public FILETIME ftCreationTime; public FILETIME ftLastAccessTime; public FILETIME ftLastWriteTime; public uint nFileSizeHigh; public uint nFileSizeLow; public int dwReserved0; public int dwReserved1; [MarshalAs(UnmanagedType.ByValTStr, SizeConst=MAX_PATH)] public string cFileName; [MarshalAs(UnmanagedType.ByValTStr, SizeConst=MAX_ALTERNATE)] public string cAlternate; } [DllImport("kernel32.dll", SetLastError = true)] static extern bool FindClose(IntPtr hFindFile); [DllImport("kernel32", CharSet=CharSet.Unicode)] private static extern IntPtr FindFirstFile( string lpFileName, out WIN32_FIND_DATA lpFindFileData); [DllImport("kernel32", CharSet=CharSet.Unicode)] private static extern bool FindNextFile( IntPtr hFindFile, out WIN32_FIND_DATA lpFindFileData); 

We do static processing. We save the metadata tree on disk and compare the stored directory tree with the loaded one, the search has been changed (based on its timestamp (faster) or file hash). In addition, we can manage deleted, added and moved, even moved files (also based on the file hash).

This implementation, mixed with the daemon that executed it every POLL_TIME, is valid for us. Hope this helps.

Answer 2

My best guess is to use the USN log, if it is a local computer, you have administrator rights and NTFS partitions. USN Magazine is extremely fast and reliable. This is a long topic, and this link explains everything: http://www.microsoft.com/msj/0999/journal/journal.aspx

Answer 3

In * nix environments, you can use inotify https://github.com/rvoicilas/inotify-tools/wiki/ , which did a great job in my limited research. There might be a version that works with windows that I have less experience ... a quick googling led me to a java clone called jnotify http://jnotify.sourceforge.net/ that advertises for working with windows, so it's worth a try .