Why is the increase in computational time when using more than four cores on this bootstrap model in R?

Question

When executing the code snippet below for the exercise of loading into R (version 2.15) on an x86_64-redhat-linux-gnu machine with 64 cores, the observed computing times seem to get worse after using more than 4 cores. Are there additional configuration changes needed for R to use more than 4 cores?

library(doMC) library(plyr) df <- empty <- data.frame(cores = numeric(), trial = numeric(), time = numeric()) for(i in 1:64){ registerDoMC(cores=i) trials <- 1000 stime <- system.time({ r <- foreach(icount(trials), .combine=cbind) %dopar% { ind <- sample(1000000, 1000000, replace=TRUE) mean(ind)}}) df<- rbind(df, data.frame(cores=i,trial=1,time=stime[3])) } df2 <- ddply(df, c("cores"), summarize, time_avg = mean(time)) 

Below is a breakdown of the processing time by the number of cores.

 num_cores time(seconds) 1 16.544 2 8.198 3 5.627 4 4.313 5 7.045 6 8.898 7 10.412 8 11.539 9 12.382 10 13.329 11 13.786 12 14.375 13 14.977 14 15.095 15 14.984 16 15.393 17 15.728 18 15.983 19 16.039 20 15.947 21 16.101 22 16.365 23 16.549 24 16.687 25 17.022 26 17.116 27 17.212 28 17.548 29 17.605 30 17.672 31 18.067 32 18.158 33 16.884 34 17.2 35 17.167 36 17.178 37 17.516 38 17.425 39 17.449 40 17.845 41 17.758 42 17.74 43 18.093 44 14.481 45 14.25 46 18.441 47 18.294 48 18.311 49 18.694 50 18.692 51 15.936 52 16.495 53 16.512 54 18.627 55 19.019 56 18.631 57 13.916 58 19.227 59 19.225 60 13.606 61 18.029 62 19.392 63 19.378 64 19.753 

Answer 1

I don’t think it’s a matter of tuning R: the tasks are so small that you cannot use many cores. From the moment of "1 core", each task takes less than 16.5 milliseconds and probably much less. The foreach package was not designed for this kind of fine-grained problem.

To have any hope, you can try using explicit chunking:

 r <- foreach(n=idiv(trials, chunks=getDoParWorkers()), .combine='c') %dopar% { sapply(seq_len(n), function(i) mean(sample(1000000, 1000000, replace=TRUE))) } 

I don't know what your goal is, but I would use mclapply directly for this problem.