How to choose chunksize for python multiprocessing with large datasets

Question

I am trying to use python to get some performance for a task that can be highly parallelized using http://docs.python.org/library/multiprocessing .

When they look at their library, they say that they use the block size for very long iterations. Now my iterable is not long, one of the dicts contained in it is: ~ 100,000 entries, with tuples as keys and numpy arrays for values.

How can I configure chunksize for this and how can I quickly transfer this data?

Thank.

Answer 1

- . multiprocessing , , , , - - , . . , , dicts . dict, , .