Geek Answers Handbook

Output `mclapply` results to data.frame

I have a very big data.frameone that I want to apply a rather complicated function to computing a new column. I want to do this in parallel. This is similar to the question posted over r listserve , but the first answer is incorrect and the second is useless.

I got everything that was found out thanks to the package parallel, except how to return the output to the data frame. Here's the MWE, which shows what I have:

library(parallel)

# Example Data
data <- data.frame(a = rnorm(200), b = rnorm(200),  
                   group = sample(letters, 200, replace = TRUE))

# Break into list
datagroup <- split(data, factor(data$group))

# execute on each element in parallel
options(mc.cores = detectCores())
output <- mclapply(datagroup, function(x) x$a*x$b)

Result in outputis a list of number vectors. I need them in the column that I can add to data. I was looking through the lines do.call(cbind, ...), but I have two lists with the same name, and not one list that I am joining. melt(output)gets one vector, but its lines do not match data.

+4
source share
5 answers

Convert from comment to response.

It works:

data <- 
  do.call(
    rbind, mclapply(
      split(data, data$group), 
       function(x){
         z <- x$a*x$b
         x <- as.data.frame(cbind(x, newcol = z))
         return(x)
         }))
rownames(data) <- seq_len(nrow(data))
head(data)
#           a          b group      newcol
#1 -0.6482428  1.8136254     a -1.17566963
#2  0.4397603  1.3859759     a  0.60949714
#3 -0.6426944  1.5086339     a -0.96959055
#4 -1.2913493 -2.3984527     a  3.09724030
#5  0.2260140  0.1107935     a  0.02504087
#6  2.1555370 -0.7858066     a -1.69383520

Since you are working with a "very large" data.frame (how rude is this?), Do you consider using dplyror data.tablefor what you are doing? For a large dataset, performance can be even better with one of them than with mclapply. The equivalent would be:

library(dplyr)
data %>%
  group_by(group) %>%
  mutate(newcol = a * b)

library(data.table) 
setDT(data)[, newcol := a*b, by=group]
+4

, .

rbind , .

unsplit.

results <- mclapply( split(data, data$group), function(x) x$a*x$b) 

resultscombined <- unsplit (results, data$group)

data$newcol <-  resultscombined

, , .

+1

parallel . , , snow .

data.frame , , c(). . :

library(snow)
library(rlecuyer)

# Example data
data <- data.frame(a = rnorm(200), b = rnorm(200),  
                   group = sample(letters, 200, replace = TRUE))
data <- data[order(data$group),]

# Cluster setup
clNode <- list(host="localhost")
localCl <- makeSOCKcluster(rep(clNode, 2))
clusterSetupRNG(localCl, type="RNGstream", seed=sample(0:9,6,replace=TRUE))
clusterExport(localCl, list=ls())

# Break into list
datagroup <- split(data, factor(data$group))

output <- clusterApply(localCl, datagroup, function(x){ x$a*x$b })

# Put back and check
data$output <- do.call(c, output)
data$check <- data$a*data$b

all(data$output==data$check)

# Stop cluster
stopCluster(localCl)
0

@beginneR dplyr, , -

 rbind_all( mclapply(split(data, data$group), fun(x) as.data.frame(x$a*x$b)))
0
source

Calculate the group average using a multi-core process:

library(dplyr)
x <- group_by(iris, Species)
indices <- attr(x,"indices")
labels <- attr(x,"labels") 

require(parallel)
result <- mclapply(indices,  function(indx){
                   data <- slice(iris, indx + 1)
                   ## Do something...
                   mean(data$Petal.Length)
                   }, mc.cores =2)

 out <- cbind(labels,mean=unlist(result))
 out
 ##      Species  mean
 ## 1     setosa 1.462
 ## 2 versicolor 4.260
 ## 3  virginica 5.552
0
source

More articles:


All Articles