Shared Network File Sharing

Question

What I want to do:

I have two folders. Each contains about 90,000 xml and bak files.

I need the xml files to sync in both folders when the file changes (of course, the new one must be copied).

The problem is this:

Due to the huge number of files and the fact that one of the directories is a shared network resource, I cannot just loop around the directory and compare the values ​​of os.path.getmtime(file) .

Even the watchdog and PyQt do not work (tried the solutions from here and here ).

Question:

Is there any other way to get an event with a modified file (on Windows systems) that works for this configuration without a loop, although all of these files?

Answer 1

So, I finally found a solution: I changed some settings of my network resource and used FileSystemWatcher

To prevent file synchronization during synchronization, I use mh5 filehash.

The code I use can be found on pastebin (This is quick and dirty code and only parts of it mentioned in the question here).

Answer 2

Code in ...

fooobar.com/questions/1465306 / ...

It seems to work for me when a network resource is being transferred.

Answer 3

I risk giving the answer that is here (you did not specify the requirements for speed, etc.), but ... Dropbox will do just that for you for free, and you won’t need to write code at all.

Of course, this may not suit your needs if you need real-time synchronization, or if you want to avoid “sharing” your files with a third party (although you can encrypt them first).

Answer 4

Can you use the second option on this page?

http://timgolden.me.uk/python/win32_how_do_i/watch_directory_for_changes.html

Answer 5

From the mention of watchdog, I assume that you are running Linux. For a local machine, inotify can help, but for a network share, you're out of luck. The extension extension for Mercurial http://hgbook.red-bean.com/read/adding-functionality-with-extensions.html has the same limitations.

In a similar situation (10K + files) I used a cloned mercury repository with inotify both on the server and on the local computer. They automatically commit and notify each other of changes. I had a slight delay (in my case there were no problems), but since benifit had a complete history of changes and it was easy to restore them after one of the systems was disconnected.