Clojure pmap / preduce vs fork-join

Question

Looks like clojure will have a fork-join implementation that looks like a functional wrapper over a java fork join framework.

I am wondering what is the difference between them and pmap / preduce?

Answer 1

Fork-join is more general than pmap / preduce , and should provide finer-grained control over parallelism. The exact APIs for this are still in the air.

Answer 2

From the look at this code, their functionality will be basically the same - the only difference is that pmap uses futures launched in Agent threadpool as a basic primitive, while pvmap uses fork-join.

I can’t say for sure, but I would expect that what works better in the general case will become the standard implementation for pmap , unless there are enough significant trade-offs to have as worthwhile,

It also looks (at least for now), the fork-join framework supports only vectors, so it is not semi-lazy like pmap .

Answer 3

These slides contain some diagrams showing comparisons between the two approaches: http://data-sorcery.org/2010/10/23/clojureconj/

Answer 4

One of the differences, as I understand it, is that pmap will only work for any degree of "chunkiness" that it sets. The function maps to each member of the sequence specified on pmap . If the grain size is too small, the potential benefits of parallelism are swallowed due to the overhead of creating and managing too many Future s.

Fork-join allows theft of work, so the number of starts on each thread can be adaptive.

Answer 5

Neither pmap nor pvmap will save us from having to use the correct block size . For my projects, which usually means breaking data into chunks and using a map on each chunk, then use pmap to match chunks in parallel. then shrink and smooth.