Dup, dup2, tmpfile and stdout in python

Question

This is the next question from here .

---

Where i want to do go

I would like to be able to temporarily redirect stdout to a temporary file, while python can still print to stdout. This will include the following steps:

• Create a copy of stdout ( new )
• Create a temporary file ( tmp )
• Redirect stdout to tmp
• Tell python to use new as stdout
• Redirect tmp to "real" stdout
• Tell python to use the "real" stdout again
• Read and close tmp

Implementation

I tried to implement the above as follows:

 import os import subprocess import sys #A function that calls an external process to print to stdout as well as #a python print to pythons stdout. def Func(s, p = False): subprocess.call('echo "{0}"'.format(s), shell = True) if p: print "print" sil = list() # <-- Some list to store the content of the temp files print "0.1" # Some testing of the Func("0.2") # functionality new = os.dup(1) # Create a copy of stdout (new) tmp = os.tmpfile() # Create a temp file (tmp) os.dup2(tmp.fileno(), 1) # Redirect stdout into tmp sys.stdout = os.fdopen(new, 'w', 0) # Tell python to use new as stdout Func("0.3", True) # <--- This should print "0.3" to the temp file and "print" to stdout os.dup2(new, 1) # Redirect tmp into "real" stdout sys.stdout = os.fdopen(1, 'w', 0) # Tell python to use "real" stdout again # Read and close tmp tmp.flush() tmp.seek(0, os.SEEK_SET) sil.append(tmp.read()) tmp.close() 

I would like to relax a bit here to take stock.
The output for the console until it is read here:

 0.1 0.2 print 

while sil should look like this: ['0.3\n'] . So everything works like a charm before here. However, if I repeat the script again like this:

 print "1.1" # Some testing of the Func("1.2") # functionality new = os.dup(1) # Create a copy of stdout (new) tmp = os.tmpfile() # Create a temp file (tmp) os.dup2(tmp.fileno(), 1) # Redirect stdout into tmp sys.stdout = os.fdopen(new, 'w', 0) # Tell python to use new as stdout # This should print "0.3" to the temp file and "print" to stdout and is the crucial point! Func("1.3", True) os.dup2(new, 1) # Redirect tmp into "real" stdout sys.stdout = os.fdopen(1, 'w', 0) # Tell python to use "real" stdout again # Read and close tmp tmp.flush() tmp.seek(0, os.SEEK_SET) sil.append(tmp.read()) 

an error occurs and the result is as follows:

 1.1 1.2 /bin/sh: line 0: echo: write error: Bad file descriptor print 

while sil reads: ['0.3\n', ''] .

In other words: the second Func("1.3", True) cannot write a temporary file.

Questions

• First of all, I would like to know why my script is not working, as if I want it to work. Sense, why is it possible only in the first half of the script to write to a temporary file?
• I am still a little puzzled using dup and dup2 . Although I think I understand how redirecting stdout to a temporary file works, I fully understand why os.dup2(new, 1) does what it does. Maybe the answer could explain that all dup and dup2 in my script do ^^

Answer 1

The reason you get a "bad file descriptor" is because the garbage collector closes the stdout FD file for you. Consider these two lines:

 sys.stdout = os.fdopen(1, 'w', 0) # from first part of your script ... sys.stdout = os.fdopen(new, 'w', 0) # from second part of your script 

Now, when the second of these two is executed, the first instance of the object object reference drops to zero, and the garbage collector destroys it. File objects close their associated fd upon destruction, and fd - 1 = stdout. Therefore, you need to be very careful how you destroy objects created with os.fdopen .

Here is a small example to show the problem. os.fstat simply used as an example function that os.fstat "Bad file descriptor" error when passing a private fd.

 import os whatever = os.fdopen(1, 'w', 0) os.fstat(1) del whatever os.fstat(1) 

Actually, I have a context manager that I think does exactly (or almost at least in my case I need a named tempfile) that you are looking for. You can see that it reuses the original sys.stdout object to avoid problems with the problem.

 import sys import tempfile import os class captured_stdout: def __init__(self): self.prevfd = None self.prev = None def __enter__(self): F = tempfile.NamedTemporaryFile() self.prevfd = os.dup(sys.stdout.fileno()) os.dup2(F.fileno(), sys.stdout.fileno()) self.prev = sys.stdout sys.stdout = os.fdopen(self.prevfd, "w") return F def __exit__(self, exc_type, exc_value, traceback): os.dup2(self.prevfd, self.prev.fileno()) sys.stdout = self.prev ## ## Example usage ## ## here is a hack to print directly to stdout import ctypes libc=ctypes.LibraryLoader(ctypes.CDLL).LoadLibrary("libc.so.6") def directfdprint(s): libc.write(1, s, len(s)) print("I'm printing from python before capture") directfdprint("I'm printing from libc before captrue\n") with captured_stdout() as E: print("I'm printing from python in capture") directfdprint("I'm printing from libc in capture\n") print("I'm printing from python after capture") directfdprint("I'm printing from libc after captrue\n") print("Capture contains: " + repr(file(E.name).read()))