Could not find function inside clusterApply

Question

I have this reproducible fragment of R

rm(list=ls()) library(doSNOW) f <- function(a, b) a+b g <- function(c) f(c*c, c+c) v <- c(1, 2, 3, 4, 5, 6) cl <- makeMPIcluster(1) cat( clusterApply(cl, v, g) ) stopCluster(cl) 

and I get the following error message:

 Error in checkForRemoteErrors(val) : 6 nodes produced errors; first error: could not find function "f" 

I am using R 2.14.1 under Ubuntu. MPI is installed and working.

I know that there is a similar problem for the foreach construct, but it allows you to refer to functions manually through the .export parameter. I could not find anything like clusterApply. Is there a workaround for this?

Thanks!

Answer 1

Your function was not sent to workers. Perhaps the best way to do this is to directly export the function:

 clusterExport(cl, list("f", "g")) 

Answer 2

I think your problem is with the "variable scope". On Mac / Linux, you have the option of using makeCluster (no_core, type = "FORK"), which automatically contains all the environment variables. On Windows, you should use the Parallel Socket Cluster (PSOCK), which starts by downloading only the base packages. This way, you always accurately determine which variables, as well as the library that you include for the parallel function to work. clusterExport () and clusterEvalQ () are needed so that the function can see the necessary variables and packages, respectively. Note that any changes to the variable after clusterExport are ignored. Get back to your problem. You should use the following:

 rm(list=ls()) library(doSNOW) f <- function(a, b) a+b g <- function(c) f(c*c, c+c) v <- c(1, 2, 3, 4, 5, 6) cl <- makeMPIcluster(1) # insert code here clusterExport(cl, list("f", "g")) # insert clusterEvalQ(cl, library(...)) if you need library for function to parallel cat( clusterApply(cl, v, g) ) stopCluster(cl)