Geek Answers Handbook

Fast reading and merging with Data.Table Fread and Rbindlist

I am looking for a way to quickly read and merge a bunch of data files using the data.table fread and rbindlist functions. I think that if fread can take a vector of file names as an argument, it could be one elegant string, like

mergeddata = rbindlist(fread(list.files("my/data/directory/")))

but since this does not look like an option, I took a more inconvenient approach to navigating through files, to read them and assign them to temporary names, and then compile a list of temporary data table names created. However, I get a crash when I try to call a list of data.table names. So my questions are: (1) how can I pass a list of data names to rbindlist in this context, and (2) in a broader sense, is there a better approach to this problem?

Thanks in advance for your time and help!

datafiles = list.files()

datatablelist = c()

for(i in 1:length(datafiles)){
  assign(paste("dt",i,sep=""),fread(datafiles[1]))
  datatablelist = append(datatablelist ,paste("dt",i,sep=""))
}

mergeddata = rbindlist(list(datatablelist))
+4
2

datatablelist = lapply(list.files("my/data/directory/"), fread), .

lapply , , , .

datatablelist = list()

for(i in 1:length(datafiles)){
  datatablelist[[datafiles[i]]] = fread(datafiles[i])
}
+4

, fread

# Load library
  library(data.table)

# Get a List of all files named with a key word, say all '.csv' files
  filenames <- list.files("C:/your/folder", pattern=glob2rx("*.csv"), full.names=TRUE)

 # Load and bind all data sets
   data <- rbindlist(lapply(filenames,fread))

, , ,

# Load data sets
  list.DFs <- lapply(filenames,fread)
+6

More articles:


All Articles