Why does my Matlab program use so much memory?

Question

I am writing a Matlab program that reads about 500 files. Each file has 20,000 lines with 1 number on each line. The program is trying to build a matrix of size 20,000 * 500 with these numbers. The numbers are stored as Binary, so 8 bytes per number. Therefore, I expect this to take 20,000 * 500 * 8 bytes, which is roughly equal to 1E8, i.e. 100 MB. And yet this program runs out of my 16 GB memory. As the program starts, I see that memory usage is steadily increasing, GB GB. I am using Matlab R2015b on Ubuntu 14.04.

What's happening? Thank you very much for your attention.

Here is the complete code

clear all; % number of rna bits in the file filesize = 20532 maxFiles = 480; rnaCounts = NaN(filesize,maxFiles); myFolder = '~/_STATS/data3/RNASeqV2/UNC__IlluminaHiSeq_RNASeqV2/Level_3'; filePattern = fullfile(myFolder, '*genes.normalized_results'); theFiles = dir(filePattern); rnaCounts = NaN(filesize,length(theFiles)); for k = 1 : length(theFiles) mrnaFilename = strtrim(theFiles(k).name); fprintf(1, 'Now reading mrnaFile %d %s \n', k, mrnaFilename); % read rna file fullFileName = fullfile(myFolder, mrnaFilename); rnafid = fopen(fullFileName); if rnafid < 0 fprintf('====ERROR OPENING RNA FILE ====================='); end rnaline = fgets(rnafid); lc = 1; % line counter while ischar(rnaline) && feof(rnafid) ~= 1 rnaline = fgets(rnafid); rnaSplit = strsplit(rnaline); % write to the matrix rnaCounts(lc,k) = str2num(rnaSplit{2}); lc = lc + 1; end fclose(rnafid); end 

Answer 1

As OP tested, the str2num function in the Linux version of Matlab 2015b has a memory leak. In any case, this function is not very useful, since it is intended for parsing strings representing entire matrices ( 1 2; 3 4 ), and not for a typical use case for parsing a single number ( 1.234 ). Use str2double for the simplest parsing of numbers; it is faster even when str2num does not break.

It is likely that using a different version of Matlab will also cost a problem, because, in my experience, these types of memory errors usually do not persist from one version to another.

Answer 2

Often high-level I / O functions, such as dlmread or textscan , are useful to read such text formats. Use dlmread if you have only numeric data, and textscan for more complex formats.

Example data you provided:

A2LD1|87769 135.5735

Since you only need the number in the second column and drop the identifier in the first column, all you have is numeric data and you can use dlmread .

 data = dlmread(fullFileName, '\t', 1, 1); 

\t indicates that the delimiter (column delimiter) is Tab. Two 1 must indicate the line offset and column offset, that is, ignore the first row (header) and first column (id) of the file.