Python: A multiprocessor card takes longer to complete the last few processes

Question

In Python, I try to run 150-200 processes. I have these 150 things in an array, and I split this array into several arrays of 10 elements each.

Now I run a multiprocessor card with 10 elements at a time. Once all 10 are complete, we will move on to the next 10, etc.

Now the problem: the ninth and tenth process is almost ALWAYS slower than the rest. Is there a reason for this? Am I not doing this in the most efficient way?

** I will not be able to share this code. So, do you have any ideas as to why this could be happening?

Thanks in advance.

Answer 1

This is due to the way Pool.mapthe data is distributed among workers. Use chunksize=1as parameter, i.e. map(...,..., chunksize=1).

A similar problem is explained in: python multiprocessor card incorrect processing of recent processes