I have a large set of tasks to run (thousands), each of which takes from 30 minutes to several hours on one processor. The memory requirements are small (a few KB). I am working on a small Linux cluster that has dozens of processors. So far, I have run them, starting several at a time, trying to manually delay the cluster.
My question is: what happens if I send hundreds or thousands at the same time - much more than the number of processors? It is clear that for each work it will take more time to start individually, but I'm interested in the overall effectiveness of this method, namely that there is exactly one task on one processor. I could also write a more sophisticated method for monitoring progress and keep each processor busy with exactly one task (for example, using multiprocessing in Python), but it will take expensive programmer time, and I wonder if the end result will actually be faster.
, , , , ( ). , - .
, - (: , , , - , , ) , . , , . , , - - , , .
, .
( ). , , , , , , , , , , .
, , Intel Thread Building Blocks, ( , ). , , , , , .. , , , , /, , .
, - (, PPL, TBB, OMP ..). , . , , , .
I/O , , , - , , . , . , , " ".
, , .
, "push", , - . , / (1 ) . - , , , .
, !
, . , . , , , , , .
, , , -say-4 *. .., 4 *.
If you use async material, it probably automatically uses tricks such as polling and thread merging, which means that it will work fine. My experience here is that asynchronous stuff usually works better than I / O threads.