Flaky file deletion under Windows 7?

Question

I have a Python test suite that creates and deletes many temporary files. On Windows 7, shutil.rmtree operations (<1% of the time) are sometimes performed. The failure is apparently random, not always in the same files, and not always the same, but it always works in rmtree operations. It seems to be a matter of time. He also recalls that Windows 7 increases vigilance with respect to access rights and administrator rights, but there are no rights (since the code just created the files) and there are no administrator rights in the mix.

It also looks like a synchronization problem between two threads or processes, but there is also no concurrency here.

Two examples of (partial) stack traces:

File "C:\ned\coverage\trunk\test\test_farm.py", line 298, in clean shutil.rmtree(cleandir) File "c:\python23\lib\shutil.py", line 142, in rmtree raise exc[0], (exc[1][0], exc[1][1] + ' removing '+arg) WindowsError: [Errno 5] Access is denied removing xml_1 File "C:\ned\coverage\trunk\test\test_farm.py", line 298, in clean shutil.rmtree(cleandir) File "c:\python23\lib\shutil.py", line 142, in rmtree raise exc[0], (exc[1][0], exc[1][1] + ' removing '+arg) WindowsError: [Errno 3] The system cannot find the path specified removing out 

In Windows XP, it never worked. On Windows 7, it fails like this in several different versions of Python (2.3-2.6, not sure about 3.1).

Has anyone seen anything like this and had a solution? The code itself is on bitbucket for a really hardworking one.

Answer 1

We had similar problems with shutil.rmtree on Windows, especially similar to your first stack trace. We solved this using an exception handler with rmtree. See this answer for more details.

Answer 2

This is a long snapshot, but do you run anything that scans directories in the background? I think antivirus / backup (maybe Windows 7 has something like built-in? I don't know). I sometimes encountered crashes when deleting / moving files from the TSVNCache.exe process that starts TortoiseSVN - it seems to look at the change directories and then probably open them to scan files.

Answer 3

I assume that you should check the code that creates the file and make sure that they are closed explicitly before proceeding to delete it. If there is nothing obvious in the code, download a copy of Process Monitor and see what happens on the file system. This tool will give you the exact error code coming from Windows and should shed light on the situation.

Answer 4

Just a thought, but if the behavior of the test (creating and deleting a large number of temporary files) is not typical of what the application really does, you can probably transfer these operations with test files to (c) StringIO and save a set of functional tests that affect The actual behavior of creating / deleting your application.

This way you can make sure that your application is behaving correctly, without the additional complexity not associated with the application.

Answer 5

That "the system cannot find the specified path": the error will appear intermittently if the path is too long for Windows (260 characters). Automated tasks often create folder hierarchies using relative links that generate fully defined paths longer than 260 characters. Any script that attempts to delete these folders using fully qualified paths will not be executed.

I built a quick workaround that used relative links on the path, but does not have suggested common code, just a warning that great answers might not help you.