Reading a large file line by line in R without a header

Question

I have a very large data file in R (in Giga). If I try to open it with R, I will get an error from memory.

I need to read the file line by line and do some analysis. I found a previous question on this issue when the file was read in n-lines and jumped to specific lines with clump. I used the answer "Nick Sabbe" and added some changes according to my needs.

Please note that I have the following test.csv file:

ABC 200 19 0.1 400 18 0.1 300 29 0.1 800 88 0.1 600 80 0.1 150 50 0.1 190 33 0.1 270 42 0.1 900 73 0.1 730 95 0.1 

I want to read the contents of a file line by line and do my analysis. So I created the following reading cycle based on the code posted by "Nick Sabbe". I have two problems: 1) The title is printed every time I print a new line. 2) Column “X” by R is also printed, although I am deleting this column.

Here is the code I'm using:

 test<-function(){ prev<-0 for(i in 1:100){ j<-i-prev test1<-read.clump("file.csv",j,i) print(test1) prev<-i } } #################### # Code by Nick Sabbe ################### read.clump <- function(file, lines, clump, readFunc=read.csv, skip=(lines*(clump-1))+ifelse((header) & (clump>1) & (!inherits(file, "connection")),1,0), nrows=lines,header=TRUE,...){ if(clump > 1){ colnms<-NULL if(header) { colnms<-unlist(readFunc(file, nrows=1, header=F)) #print(colnms) } p = readFunc(file, skip = skip, nrows = nrows, header=FALSE,...) if(! is.null(colnms)) { colnames(p) = colnms } } else { p = readFunc(file, skip = skip, nrows = nrows, header=header) } p$X<-NULL # Note: Here I'm setting the index to NULL return(p) } 

The output I get is:

  ABC 1 200 19 0.1 NA 1 1 1 1 2 400 18 0.1 NA 1 1 1 1 3 300 29 0.1 NA 1 1 1 1 4 800 88 0.1 NA 1 1 1 1 5 600 80 0.1 

I want to get rid of the rest of the reading:

  NA 1 1 1 

Also, is there a way to make a for stop loop when the end of the file is such an EOF in another language ???

Answer 1

Perhaps something like this might help you:

 inputFile <- "foo.txt" con <- file(inputFile, open = "r") while (length(oneLine <- readLines(con, n = 1)) > 0) { myLine <- unlist((strsplit(oneLine, ","))) print(myLine) } close(con) 

or by scanning to avoid splitting like @MatthewPlourde

I use scan: I skip the header, and quiet = TRUE so that there is no message about how many elements there were

 while (length(myLine <- scan(con,what="numeric",nlines=1,sep=',',skip=1,quiet=TRUE)) > 0 ){ ## here I print , but you must have a process your line here print(as.numeric(myLine)) }