Geek Answers Handbook

What is the fastest way to load data into matlab

I have a huge amount of data (> 800 MB) that takes the age to load into Matlab mainly because it breaks into tiny files every <20kB. They are all in a proprietary format that I can read and load into Matlab, it just took so long.

I am going to read the data and write it to some kind of binary file, which should make it faster for subsequent readings (which can be many, so I need acceleration).

So my question is: what is the best format to write to disk to read them as quickly as possible?

I assume that I have the ability to write using fwrite or just save variables from matlab. I think I would prefer the fwrite option, so if necessary, I could read them from another package / language ...

+5
source share
3 answers

Look at the HDF5 data format used in recent versions of MATLAB as the main format for .mat files. You can manually create your own HDF5 files using the function hdf5write, and this file can be accessed from any language with HDF bindings (the most common languages, or at least offer a way to integrate C code that can call the HDF5 library).

( ), (fwrite).

+4

.

save myfile.mat <var_a> <var_b> ...
+2

, -v6 .mat :

save(matlabTrainingFile, 'Xtrain', 'ytrain', '-v6');

, ...

Attr Name                   Size                     Bytes  Class
==== ====                   ====                     =====  ===== 
  g  Xtest               1430x4000                45760000  double
  g  Xtrain              3411x4000               109152000  double
  g  Xval                1370x4000                43840000  double
  g  ytest               1430x1                      11440  double
  g  ytrain              3411x1                      27288  double
  g  yval                1370x1                      10960  double

... , :

:

time to load the training data: 78 SECONDS!!! 
time to load validation data:   32
time to load the test data:     35

:

time to load the training data: 0 SECONDS!!!
time to load validation data:   0
time to load the test data:     0

-, , , , 6 , . , , WAY.

0

More articles:


All Articles