Why is the memory usage for the child process (python multiprocessing) so different when sharing ctypes.Structure with a string or just a string?

Question

The following code uses multiprocessing Array to share a large array of unicode strings for all processes. If I use c_wchar_p as a type, memory usage for the child process is about a quarter of the memory used in the parent process (the amount changes if I change the number of records in the array).

However, if I use ctypes.Structure with one c_wchar_p field, the memory usage for the child process is constant and very low, while the memory usage of the parent process is doubled.

 import ctypes import multiprocessing import random import resource import time a = None class Record(ctypes.Structure): _fields_ = [('value', ctypes.c_wchar_p)] def __init__(self, value): self.value = value def __str__(self): return '(%s)' % (self.value,) def child(i): while True: print "%ik memory used in child %i: %s" % (resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024, i, a[i]) time.sleep(1) for j in xrange(len(a)): c = a[j] def main(): global a # uncomment this line and comment the next to switch #a = multiprocessing.Array(ctypes.c_wchar_p, [u'unicode %r!' % i for i in xrange(1000000)], lock=False) a = multiprocessing.Array(Record, [Record(u'unicode %r!' % i) for i in xrange(1000000)], lock=False) for i in xrange(5): p = multiprocessing.Process(target=child, args=(i + 1,)) p.start() while True: print "%ik memory used in parent: %s" % (resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024, a[0]) time.sleep(1) if __name__ == '__main__': main() 

Using c_wchar_p leads to this conclusion:

 363224k memory used in parent: unicode 0! 72560k memory used in child 5: unicode 5! 72556k memory used in child 3: unicode 3! 72536k memory used in child 1: unicode 1! 72568k memory used in child 4: unicode 4! 72576k memory used in child 2: unicode 2! 

Using the recording results on this output:

 712508k memory used in parent: (unicode 0!) 1912k memory used in child 1: (unicode 1!) 1908k memory used in child 2: (unicode 2!) 1904k memory used in child 5: (unicode 5!) 1904k memory used in child 4: (unicode 4!) 1908k memory used in child 3: (unicode 3!) 

Why?

Answer 1

I don't know about increasing memory usage, but I don't think it really does what you intend to do.

If you change a[i] in the parent process, the child processes do not get the same value.

It is best not to skip pointers (such types of _p ) between processes. As stated in multiprocessing docs:

Although you can store a pointer in shared memory, remember that this will refer to a location in the address space of a specific process. However, the pointer is likely to be invalid in the context of the second process, and attempting to dereference the pointer from the second process may result in failure.