Dlmread returns one column for large text files

Question

I am trying to read in a large file with dlmread , but it just treats the whole file as one long column. The file is written in Java with the following code:

 public void writeToFile(double[] arr) throws IOException{ FileWriter write = new FileWriter(path, append); PrintWriter print_line = new PrintWriter(write); for(int i=0; i<arr.length; i++){ print_line.printf("%f\t", arr[i]); } print_line.printf("\n"); print_line.close(); } 

and my MATLAB script reads in a file like: [DATA] = dlmread('probability_cyclelength.dat'); , Giving:

 >>size(DATA) ans = 2000000 1 

A total of 2,000,000 data, with up to 60,000 per line (but not the same number in each line) should not matter.

When I try to use it with a smaller dataset (100,000 data), it works absolutely fine. I don’t know the problem is in Java or MATLAB, so I really need help, thanks!

Answer 1

By default, dlmread tries to dlmread delimiter from the file, by default it uses a space as the delimiter.

The only way I was able to reproduce the problem you described is to specify ' ' as a separator. Are you sure you are not doing this?

Try making this change and see if it fixes your problem.

 data = dlmread(inFile, '\t'); 

If this does not fix your problem, I suspect that the problem arises because the lines in your text file have different numbers of columns. For example, if you use dlmread to open a text file containing:

 1 2 3 4 5 

dlmread returns such a matrix:

 1 2 3 4 5 0 0 0 

This view is useless because it uses 64 bytes (8 bytes per double * 8 double) to store 40 bytes of information.

It is possible that with these empty positions, the matrix representation of your file is simply too large, so dlmread returns your vector instead of saving memory.

You can get around this though. If you only need a few lines at a time, you can load a collection of lines from a file by specifying range to dlmread . Please note that for this you need to know the maximum number of columns in the file, since dlmread will not allow you to read more than this number of columns.

 r = [0 4]; %load the first 5 rows maxC = 10; % load up to 10 columns data = dlmread(inFile, '\t', [r(1), 0, r(2), maxX]); 

You can then scroll through the file by loading the lines of interest, but you probably cannot load them all into the matrix due to the memory limitations that I mentioned earlier.

If you need the entire data set in memory, you should consider loading each row individually and storing them in an array of cells. It takes a bit more work to get everything loaded, but you can do something like this:

 % open the file fid = fopen(fileName); % load each line as a single string tmp = textscan(fid, '%s', 'delimiter', '\n'); % textscan wraps its results in a cell, remove that wrapping rawText = tmp{1}; nLines = numel(rawText); %create a cell array to store the processed string data = cell(nLines, 1); for i = 1:nLines %scan a line of text returning a vector of doubles tmp = textscan(rawText{i}, '%f'); data{i} = tmp{1}; end