The Tempfile module and Python threads do not play well; What am I doing wrong?

Question

I had an interesting problem with threads and the tempfile module in Python. It seems that something is not being cleared until the threads come out, and I am running against the limit of the open file. (This applies to OS X 10.5.8, Python 2.5.1.)

However, if I replicate what the tempfile module does (not all security checks, it just generates a file descriptor and then using os.fdopen to create the file) I have no problem.

Before writing this as an error with Python, I thought that I would check here, since it is much more likely that I am doing something inaccurate. But if I, then the day, trying to understand this, did not deliver me anywhere.

#!/usr/bin/python import threading import thread import tempfile import os import time import sys NUM_THREADS = 10000 def worker_tempfile(): tempfd, tempfn = tempfile.mkstemp() tempobj = os.fdopen(tempfd, 'wb') tempobj.write('hello, world') tempobj.close() os.remove(tempfn) time.sleep(10) def worker_notempfile(index): tempfn = str(index) + '.txt' # The values I'm passing os.open may be different than tempfile.mkstemp # uses, but it works this way as does using the open() function to create # a file object directly. tempfd = os.open(tempfn, os.O_EXCL | os.O_CREAT | os.O_TRUNC | os.O_RDWR) tempobj = os.fdopen(tempfd, 'wb') tempobj.write('hello, world') tempobj.close() os.remove(tempfn) time.sleep(10) def main(): for count in range(NUM_THREADS): if count % 100 == 0: print('Opening thread %s' % count) wthread = threading.Thread(target=worker_tempfile) #wthread = threading.Thread(target=worker_notempfile, args=(count,)) started = False while not started: try: wthread.start() started = True except thread.error: print('failed starting thread %s; sleeping' % count) time.sleep(3) if __name__ == '__main__': main() 

If I started it using the worker_notempfile active line and the worker_tempfile line, it ends.

In another way (using worker_tempfile ) I get the following error:

 $ python threadtempfiletest.py Opening thread 0 Opening thread 100 Opening thread 200 Opening thread 300 Exception in thread Thread-301: Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/threading.py", line 460, in __bootstrap File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/threading.py", line 440, in run File "threadtempfiletest.py", line 17, in worker_tempfile File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/tempfile.py", line 302, in mkstemp File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/tempfile.py", line 236, in _mkstemp_inner OSError: [Errno 24] Too many open files: '/var/folders/4L/4LtD6bCvEoipksvnAcJ2Ok+++Tk/-Tmp-/tmpJ6wjV0' 

Any ideas what I'm doing wrong? Is this a bug in Python, or am I with a bone?

UPDATE 2009-12-14: It seems I found an answer, but I don't like it. Since no one could reproduce the problem, I went hunting around our office for cars. He passed everything except my car. I tested on a Mac with the same software versions that I used. I even went in search of the G5 desktop with the EXACT same hardware and software that I had - the same result. Both tests (with tempfile and without tempfile) succeeded in everything.

For kicks, I downloaded Python 2.6.4 and tried it on my desktop, and the same template on my system as Python 2.5.1: tempfile failed, and the notemp file was successful.

This leads me to the conclusion that something was on my Mac, but I definitely can not understand what. Any suggestions are welcome.

Answer 1

Since no one could reproduce the problem, I went hunting around our office for cars. He passed everything except my car. I tested on a Mac with the same software versions that I used. I even went in search of the G5 desktop with the EXACT same hardware and software that I had - the same result. Both tests (with tempfile and without tempfile) succeeded in everything.

For kicks, I downloaded Python 2.6.4 and tried it on my desktop, and the same template on my system as Python 2.5.1: tempfile failed, and the notemp file was successful.

This leads me to the conclusion that something was on my Mac, so this is unlikely to be the problem anyone else is having.

Thanks to VERY everyone for everyone (especially Alex Martelli) who helped on this!

Answer 2

I cannot reproduce the problem with (Apple's own property) Python 2.5.1 on Mac OS X 10.5.9 - it works just fine until completion!

I tried both on the Macbook Pro, i.e. on the Intel processor, and on the old PowerMac, i.e. on the PPC processor.

Therefore, I can only imagine that in 10.5.8 there was an error that I never noticed (I do not have 10.5.8 for testing, as I always update quickly whenever it offers a software update). All I can offer is that you are trying to upgrade to 10.5.9 and see if the error disappears - if it’s not, I don’t know how this difference in behavior between my machines and yours is possible.

Answer 3

I think your answer can be found here . You must explicitly specify the os.close() file descriptor specified as the first part of the tuple that mkstemp gives you.

Edit: no, the OP is already doing what needs to be done. I leave a response for a good link.

Answer 4

I just checked your code on my Ubuntu Linux machine here and it worked fine for me.

I have one suggestion for you to try. I do not know that this will help, but it will not hurt. Rewrite your code for use with:

 from __future__ import with_statement def worker_tempfile(): tempfd, tempfn = tempfile.mkstemp() with os.fdopen(tempfd, 'wb') as tempobj: tempobj.write('hello, world') os.remove(tempfn) time.sleep(10) 

The with statement must ensure that the file object is closed, no matter what. Maybe this can help?

Good luck. Great job on this, by the way.

Answer 5

Why do you think the error is not genuine? You start 10,000 threads, each of which opens a file, while the maximum number of open files is usually 1024 on Unix systems.

First, try manually tracking the number of files currently open and checking to see if it exceeds the OS limit.