Initializing and destroying Python workers for multiprocessing

Question

I have a model that I call many times from Python. The model takes a lot of time to start and stop, but only a short time to process the input data (which can be done several times between start / shutdown). Multiprocessing pool () seemed like a good way to do this, but I had trouble getting the Model () class to destroy correctly.

The following is a simplified structure of the program code. In fact, the init and del functions should do some smart things with the win32com.client module, and the model.y variable is a handle to control the external application.

#!/usr/bin/env python import multiprocessing import random import time class Model(): def __init__(self): self.y = random.randint(0,5) # simplification time.sleep(5) # initialisation takes a while... def __del__(self): del(self.y) # simplification time.sleep(1) # destruction isn't especially quick either def init(): global model model = Model() def f(x): # worker function, runs quickly return model.y * x ** 2 def main(): pool = multiprocessing.Pool(processes=2, initializer=init) it = pool.imap(f, range(4)) for n in range(4): print it.next() pool.close() if __name__ == '__main__': main() 

The del function is never called for Model () s, I assume the garbage collector contains some reference. How can I make sure the model is closed correctly at the end of the program?

Answer 1

The johnthexiii solution will kill the model when the working function is first run. You can offer a separate destruction function:

 import time def g(x): # destroy function del model time.sleep(1) # to ensure, this worker does not pick too many return None 

Before pool.close() you add

 pool.map_async(g, range(2), 1) # if specified to have two processes before 

I do not think this is a very "clean" solution, but it should work.

Answer 2

What about

 def f(x): # worker function, runs quickly y = model.y del model return y * x ** 2