How to pass file descriptors from parent to child in python?

Question

I use a multiprocessor module and using pools to run multiple employees. But file descriptors that open in the parent process are closed in workflows. I want them to be open ..! Is there a way to pass file descriptors for sharing between parents and children?

Answer 1

I do not know how to exchange file descriptors between processes. If there is a way, it most likely depends on the OS.

I assume that you need to exchange data at a different level.

Answer 2

You can use _multiprocessing.sendfd/recvfd to transfer file descriptors between processes:

Code example:

 # Before fork child_pipe, parent_pipe = multiprocessing.Pipe(duplex = True) # Main process import _multiprocessing # has socket_to_pass socket object which want to pass to the child _multiprocessing.sendfd(parent_pipe.fileno(), socket_to_pass.fileno()) # Worker process import _multiprocessing import socket import os fd = _multiprocessing.recvfd(self.child_pipe.fileno()) # rebuild the socket object from fd received_socket = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM) # socket.fromfd() duplicates fd, so we can close the received one os.close(fd) # and now you can communicate using the received socket received_socket.send("hello from the worker process\r\n") 

Unfortunately, this module is not documented in the Python help system.

Answer 3

There is also a multiprocessing fork called multiprocess , which replaces pickle with dill . dill can distinguish file descriptors, and thus multiprocess can easily transfer them between processes.

 >>> f = open('test.txt', 'w') >>> _ = map(f.write, 'hello world') >>> f.close() >>> import multiprocess as mp >>> p = mp.Pool() >>> f = open('test.txt', 'r') >>> p.apply(lambda x:x, f) 'hello world' >>> f.read() 'hello world' >>> f.close()