Files that appear in a directory faster than I can process them (pyinotify)

Question

I am using pyinotify (event process_IN_MOVED) to view the files displayed in the directory. When a new file appears, this event will be fired and the file will be processed in some way. The problem is that sometimes files appear in the directory faster than I can process them, which means that a bunch of files will not be processed. I may have a function to sleep for ten seconds and then wake up to find new files or something like that, but I really want to stick with an event-based solution, if possible. Is there any way to do this?

Answer 1

Typically, I would implement a thread pool here to handle the processing, while an event observer would just watch the events and pass them to the pool. Example:

(event happens) -> Watcher registers the event -> puts it into the thread pool queue -> thread pool processes the event 

Thus, the observer will spend minimal time outside the waiting part, thereby significantly reducing the likelihood of skipping the update.

Answer 2

Twisted has inotify support . You can give it a callback to handle. You definitely do not want to be sleep ing. It depends on what processing you do, whether you want to do it in the process or through another process, but you should not lose events.

Answer 3

As Humungus notes, a thread pool is a good option.

I just wrote the code that I wrote for this problem at:

https://github.com/timstaley/autocrunch

It is a little cluttered with application-specific details, but you should be able to perform replacement functions for your needs. At some point, I could write a blog post with an abridged version, but now I do not have time. NTN!