Python file objects, closures, and destructors

Question

Description tempfile.NamedTemporaryFile() says:

If delete is true (the default), the file is deleted as soon as it closes.

In some cases, this means that the file is not deleted after the Python Translator finishes. For example, when performing the following test in py.test , the temporary file remains:

 from __future__ import division, print_function, absolute_import import tempfile import unittest2 as unittest class cache_tests(unittest.TestCase): def setUp(self): self.dbfile = tempfile.NamedTemporaryFile() def test_get(self): self.assertEqual('foo', 'foo') 

In a way, this makes sense because this program never explicitly closes the file. The only way to close the object is supposed to be in the __del__ destructor, but here the link language claims that "it is not guaranteed that the __del__() methods for objects that still exist when the translator exits. Everything is still consistent with the documentation.

However, I am confused about the consequences of this. If it’s not that the file’s objects are closed at the interpreter’s output, can it possibly be that some data that was successfully written to the (buffered) file object is lost, although the program exits gracefully because it was still in the object object buffer, and the object the file never closed?

Somehow it seems very unlikely and non-pythic to me, and open () in the documentation also does not have such warnings. So I (tentatively) conclude that file objects are, after all, guaranteed to be private.

But how does this magic happen, and why can't NamedTemporaryFile() the same magic to make sure the file is deleted?

Edit: note that I'm not talking about file descriptors here (which the OS buffers and the OS closes when the program exits), but Python files that can implement their own buffering.

Answer 1

On Windows, NamedTemporaryFile uses the Windows extension (os.O_TEMPORARY) to ensure that the file will be deleted when it is closed. This probably also works if the process is killed in any way. However, on POSIX there is no obvious equivalent, most likely because on POSIX you can simply delete files that are still in use; it only deletes the name, and the contents of the file are deleted only after it is closed (in any way). But on the condition that we want the file name to be preserved until the file is closed, for example, using NamedTemporaryFile, then we need "magic".

We cannot use the same magic as for cleaning buffered files. What happens is that the C library processes it (in Python 2): the files are FILE objects in C, and C ensures that they are flushed to the normal output of the program (but not if the process is killed). In the case of Python 3, there is special C code to achieve the same effect. But this is specific to this use case, and not to something that can be reused.

This is why NamedTemporaryFile uses a custom __del__ . And indeed, __del__ not guaranteed when a translator is called. (We can prove this with a global link loop that also references an instance of NamedTemporaryFile or runs PyPy instead of CPython.)

As a side note, NamedTemporaryFile can be implemented a little more reliably, for example. logging in with atexit to make sure the file name is deleted. But you can also call it: if your process does not use an unlimited number of NamedTemporaryFiles, it is just atexit.register(my_named_temporary_file.close) .

Answer 2

In any version of * nix, all file descriptors are closed when the process ends, and the operating system takes care of this. In this respect, Windows is probably exactly the same. Without delving into the source code, I cannot say with 100% authority what is actually happening, but it is likely what is happening:

• If delete is False , unlink() is called (or a similar function on other operating systems). This means that the file will be automatically deleted when the process ends, and there are no more open file descriptors. While the process is running, the file will still remain.

• If delete is True , the C remove() function is probably used. This will force the file to be deleted before the process exits.

Answer 3

File buffering is handled by the operating system. If you do not close the file after opening it, this is because you assume that the operating system will clear the buffer and close the file after the owner exists. This is not Python magic; it is your OS that does this. The __del__() method is Python related and requires explicit calls.