Geek Answers Handbook

Reading from stdin in Python Process?

I am trying to read from sys.stdin from inside a Python Process object, but I keep getting the result "ValueError: I / O operation on closed file". Here is a quick example:

import sys from multiprocessing import Process def do_something(input_data): for x in input_data: print x input=sys.stdin p = Process(target=do_something, args=(input,)) p.start() p.join() #Wait for Process to complete

The above script always fails:

 Traceback (most recent call last): File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "example.py", line 6, in do_something for x in input_data: ValueError: I/O operation on closed file

Just calling do_something(input) works fine without using a process, of course. Creating a Pipe() object seems to help - I can write the contents of stdin in Pipe and get the results in string form from the process, but I really need to enter the file form for some subsequent operations. I could dump the contents into a file and re-read it from inside the Process, but this seems pretty awkward, especially if the stdin is really big. Is there an easy way to read from sys.stdin from inside the process?

+1
source share
2 answers

This is because stdin closes before starting the process. Otherwise, it may happen that both the parent and the child processes (or several child processes) try to read from the same stdin, which is a bad idea.

In the child process, sys.stdin actually redirected to /dev/null :

 from multiprocessing import Process import sys def test(*args): print(args) print(sys.stdin, sys.stdin.fileno()) if __name__ == '__main__': p = Process(target=test, args=(sys.stdin,)) p.start() p.join()

should print something similar to this:

 (<closed file '<stdin>', mode 'r' at 0x7f3b4564b0c0>,) (<open file '/dev/null', mode 'r' at 0x7f3b43a9e0c0>, 3)

The past argument here is a reference to a private file object, trying to use it will result in the error you saw.

You can get around this by using os.dup() in sys.stdin.fileno() in the parent object and pass the returned copy of the file descriptor to child as an argument, where you can use os.fdopen() to work with it.

A cleaner solution would probably be to read the input in the parent process and pass it to the child using multiprocessing.Queue .

+2
source

You must close the file you are trying to write at some point. Check your code and try to delete all lines closing the files (file VariableName.close ()) and see if it works. If so, repeat adding them one by one to find the problem. After you find the line causing the problems, try moving it further into the program (call it later) and see if this fixes your problems.

EDIT: change

 def do_something(input_data): for x in input_data: print x

to

 def do_something(): for x in sys.stdin: print x

and get rid of input = sys.stdin

0
source

More articles:


All Articles