When copied when writing called for python multiprocessing, in class methods

Question

I have some multiprocessing Python code that looks something like this:

 import time from multiprocessing import Pool import numpy as np class MyClass(object): def __init__(self): self.myAttribute = np.zeros(100000000) # basically a big memory struct def my_multithreaded_analysis(self): arg_lists = [(self, i) for i in range(10)] pool = Pool(processes=10) result = pool.map(call_method, arg_lists) print result def analyze(self, i): time.sleep(10) return i ** 2 def call_method(args): my_instance, i = args return my_instance.analyze(i) if __name__ == '__main__': my_instance = MyClass() my_instance.my_multithreaded_analysis() 

After reading answers about how memory works in other StackOverflow answers, such as Using Python multiprocessing memory , it seemed to me that this would not use memory proportionally to how many threads I used for multiprocessing, since this is copying and writing, but I did not change none of my_instance attributes. However, I see high memory for all processes, when I run the top, this indicates that most of my processes use a lot of memory (this is the top output from OSX, but I can replicate to Linux).

My question is basically, am I interpreting this correctly, since my instance of MyClass actually duplicated through the pool? And if so, how can I prevent this; Should I just not use such a design? My goal is to reduce memory usage for computational analysis.

 PID COMMAND %CPU TIME #TH #WQ #PORT MEM PURG CMPRS PGRP PPID STATE 2494 Python 0.0 00:01.75 1 0 7 765M 0B 0B 2484 2484 sleeping 2493 Python 0.0 00:01.85 1 0 7 765M 0B 0B 2484 2484 sleeping 2492 Python 0.0 00:01.86 1 0 7 765M 0B 0B 2484 2484 sleeping 2491 Python 0.0 00:01.83 1 0 7 765M 0B 0B 2484 2484 sleeping 2490 Python 0.0 00:01.87 1 0 7 765M 0B 0B 2484 2484 sleeping 2489 Python 0.0 00:01.79 1 0 7 167M 0B 597M 2484 2484 sleeping 2488 Python 0.0 00:01.77 1 0 7 10M 0B 755M 2484 2484 sleeping 2487 Python 0.0 00:01.75 1 0 7 8724K 0B 756M 2484 2484 sleeping 2486 Python 0.0 00:01.78 1 0 7 9968K 0B 755M 2484 2484 sleeping 2485 Python 0.0 00:01.74 1 0 7 171M 0B 594M 2484 2484 sleeping 2484 Python 0.1 00:16.43 4 0 18 775M 0B 12K 2484 2235 sleeping 

Answer 1

Everything sent to pool.map (and the methods associated with it) does not actually use shared resources for copying to write. The values "pickled" (Python serialization mechanism) are piped to workflows and scattered there, which reconstructs the object in the child from the scratch. Thus, each child in this case ends with a version of the source data with a copy to the record (which he never uses, since he was told to use a copy sent via IPC), and a personal recreation of the source data that was reconstructed in the child and is not used.

If you want to take advantage of copying on the copy side, you cannot send data (or objects that reference data) through the pipe. You must store them in a place that can be found in the child by accessing their own global variables. For example:

 import time from multiprocessing import Pool import numpy as np class MyClass(object): def __init__(self): self.myAttribute = np.zeros(100000000) # basically a big memory struct def my_multithreaded_analysis(self): arg_lists = list(range(10)) # Don't pass self pool = Pool(processes=10) result = pool.map(call_method, arg_lists) print result def analyze(self, i): time.sleep(10) return i ** 2 def call_method(i): # Implicitly use global copy of my_instance, not one passed as an argument return my_instance.analyze(i) # Constructed globally and unconditionally, so the instance exists # prior to forking in commonly accessible location my_instance = MyClass() if __name__ == '__main__': my_instance.my_multithreaded_analysis() 

Without skipping self , you avoid making copies and simply use the only global object that has been copied to the record that is mapped to the child. If you need more than one object, you can do a global mapping of list or dict object instances before pooling, then pass an index or key that can search for the object as part of argument (s) to pool.map . Then the working function uses the index / key (which should have been pickled and sent to the child via IPC) to search for the value (copy to write) displayed in the global dict (also displayed copy to write) so you copy cheap information to search for expensive data in a child without copying.