How to know when to manage resources in Python

Question

I hope I correctly formulated the question. I try to force myself to be a better programmer. For the better, I mean effective. I want to write a program to identify files in a directory and read each file for further processing. After some shuffling, I got to this:

for file in os.listdir(dir): y=open(dir+'\\'+file,'r').readlines() for line in y: pass y.close() 

No wonder I get the AttributeError attribute since y is a list. I did not think about it when I wrote the fragment.

I think about it and am afraid that I have five open files (there are five files in the directory indicated by dir.

I can fix the code so that it works, and I immediately close the files after they open. I am wondering if I need to, or if Python handles closing the file in the next loop iteration. If so, I only need to write:

 for file in os.listdir(dir): y=open(dir+'\\'+file,'r').readlines() for line in y: pass 

I assume that he (python) handles this effortlessly. The reason I think this can be handled is because I changed the object / thing referenced by y. When I start the second iteration, there are no more links to the file opened and read using the readlines method.

Answer 1

Python closes open files when they collect garbage, so you can usually forget about it - especially when reading.

However, if you want to explicitly close, you can do this:

 for file in os.listdir(dir): f = open(dir+'\\'+file,'r') y = f.readlines() for line in y: pass f.close() 

However, we can immediately improve this, because in python you can directly iterate over object files:

 for file in os.listdir(dir): y = open(dir+'\\'+file,'r') for line in y: pass y.close() 

Finally, in recent python, there is a with statement:

 for file in os.listdir(dir): with open(dir+'\\'+file,'r') as y: for line in y: pass 

When the with block ends, python will close the file for you and clear it.

(you might also want to look into os.path for more pythonic tools for managing file names and directories)

Answer 2

Do not worry about it. The Python garbage collector is good, and I never had a problem closing file pointers (for read operations at least)

If you want to explicitly close the file, just save open() in one variable, and then call readlines() on this, for example ..

 f = open("thefile.txt") all_lines = f.readlines() f.close() 

Or you can use the with statement, which was added in Python 2.5 as an import from __future__ and "correctly" added in Python 2.6:

 from __future__ import with_statement # for python 2.5, not required for >2.6 with open("thefile.txt") as f: print f.readlines() # or the_file = open("thefile.txt") with the_file as f: print f.readlines() 

The file will be automatically closed at the end of the block.

.. but there are other more important things to worry about in the snippets you posted, mostly stylistic things.

First, try to avoid manually constructing paths by concatenating strings. The os.path module contains many ways to make this a more reliable cross-platform way.

 import os y = open(os.path.join(dir, file), 'r') 

In addition, you use two variable names, dir and file - both of which are built-in functions. Pylint is a good tool for detecting such things, in which case it will give a warning:

 [W0622] Redefining built-in 'file'