FileSystemWatcher for notification

Question

I am creating a Windows application that uses FileSystemWatcher. FileSystemWatcher oversees some change directory. And every time a file is added to this directory, FileSystemWatcher must add information about this file to the XML file. Everything works fine, but when I add, for example, 100 files at the same time (say that some application adds these files to the directory), not every file information appears in this XML file.

I want to use Queue. And use it to add items to this collection. And use a timer. The timer will add information to the XML from this collection. Is that a good idea?

Can anyone advise me what to do?

---

So, I think I should create a Windows application and a Windows service. WinApp will only add information to EventLog, and the Windows service will read the EventLog information and write it to XML. I think this is the best way to do this. I am waiting for good advice.

Answer 1

As Michael Stam writes in his answer, you can try increasing the size of the buffer ( FileSystemWatcher.InternalBufferSize ). Please note, however, that you should not set this value to a value too high. In addition, IMHO is perhaps only a temporary fix, see What happens when you add even more files to your folder at the same time.

I read about some other things you can try if increasing the size of the buffer does not help:

• If you subscribe to FileSystemWatcher notification events, try to process the event handler as short as possible; i.e. make sure the execution does not last long. If you need to do a lot of work for file notification, you can try to start a separate thread and do the processing there; your event handler could return to the caller very quickly (and the file notification will be removed from the buffer / queue sooner).

• Do not use the additional information provided by FileSystemWatcher other than the basic notification that something has changed. As soon as you receive a file change notification, wait a short time until more notifications are received (i.e., wait for the last notification of 100 simultaneous file change notifications). Then list the directory contents manually and pass the necessary information into your XML.

  This has one major drawback: manually determining whether files have been deleted, renamed or created is not easy. Your program would have to save the last list of directories with which the current list can be compared to find out what has changed.

  The advantage of this is that you can be sure that no changes will be deleted due to the FileSystemWatcher buffer overflow.

Answer 2

FileSystemWatcher has an internal buffer for changes. When many quick changes occur, the buffer may not capture all events.

Try increasing the InternalBufferSize to something higher.

You can set the buffer to 4 KB or more, but it should not exceed 64 KB. For best performance, use multiple 4K computers on Intel-based computers.

The system notifies the component of file changes and saves these changes in the buffer that the component creates and passes to the API. Each event can use up to 16 bytes of memory, not counting the file name. If many changes occur in a short time, the buffer may overflow. This causes the component to lose track of changes in the directory, and it will provide only a hidden notification. An increase in buffer size has the following consequences:

An increase in buffer size can prevent the absence of file system change events. Please note that an instance of the FileSystemWatcher class may raise an Error event when an event is skipped or the buffer size is exceeded due to dependencies with the Windows operating system.

Increasing the size of the buffer is expensive because it comes from inexperienced memory, which cannot be replaced with a disk, so keep the buffer as small as possible. To avoid buffer overflows, use the NotifyFilter and IncludeSubdirectories properties to filter out unwanted change notifications.

For diagnostics, perhaps you should first subscribe to the Error Event to see if this is really a buffer overflow.

Also, as said, set NotifyFilter to the smallest required flags, which can be NotifyFilters.LastWrite if you want to track changes.

Answer 3

There is a critical note in MSDN docs that can help you more reliably detect changes:

Keep your event handling code as short as possible.

I suspect (but I don’t know for sure) that this is due to the fact that file system events are raised in the main observer stream, so at any time spent processing events, a window is created in which changes can go unnoticed. As you described your solution, it seems likely that you are doing I / O in the callback (writing to the XML change log file), and this can definitely be too much work to do built-in, according to the criteria of the API Docs. If you have a lot of work to do with an event, release your events for processing in a separate thread so that you can return to viewing the file system as soon as possible.

One relatively easy way to do this is to use ThreadPool.QueueUserWorkItem . This means that your change log will still not be 100% synchronized with the state of the file system (due to the delay caused by using separate threads and queues), but this may be more accurate and seems to be your main problem. You need to make sure that your WaitCallback called by the threadpool file is thread safe (for example, writing to the change log is not performed simultaneously without lock() or the like), and keep in mind that there is no guarantee that change log entries will be written to the order in which they occurred (although it is doubtful that FileSystemWatcher guarantees this anyway).

Also in accordance with the recommendations of the API - make sure that you are filtering a possible set of events, so you only get a callback for those that you absolutely must see:

To avoid buffer overflows, use the NotifyFilter and IncludeSubdirectories properties buttons so you can filter out notifications of unwanted changes.

Answer 4

If I understand correctly: you look at the directory, and when many files are added at the same time, you see only some of them with FileSystemWatcher .

Have you tried the following: in the event handler for OnCreated, just go to the file system and get the full contents of the directory, regardless of what the event tells you?