Long data loading time in Matlab

Question

I have four variables, each of which is stored in 365 matte files (size: 8 x 92 x 240). I am trying to load them into my function on cycle day = 1: 365, one variable file per day. However, the first two variables always take an abnormally long time to load. My download code is as follows:

load([eraFolder sprintf('Y%dD%d-tempSD.mat',year,day)], 'tempSD'); % took 5420 s to load load([eraFolder sprintf('Y%dD%d-tempDewSD.mat',year,day)], 'tempDewSD') load([eraFolder sprintf('Y%dD%d-eEraSD.mat',year,day)], 'eEraSD'); % took 6 seconds to load load([eraFolder sprintf('Y%dD%d-pEraSD.mat',year,day)], 'pEraSD'); 

Using Profiler, I could see that the first two variables took 5420 seconds to load 365 calls, while the last two variables took 6 and 4 seconds to load more than 365 calls, respectively. When I change the order in which variables are loaded, for example. eEraSD to tempSD , still the first two loads take longer.

When using tic toc to track the time taken to load, it seems that the load time of the first or second variable exponentially increases with the number of calls (for the last calls that take 50 seconds to start). For the third and fourth variables, the load time remains around 0.02 -0.04 seconds per file, more or less independent of how far in the for loop I have gone. See the figures below.

When using importdata instead of "load", the first line takes about 8000 seconds to load 365 times (with an exponential increase in load, as shown for T in the second figure). The remaining lines take about 10 seconds to load 365 times.

I can’t understand why it looks like this and what I can do to reduce load time. I would really appreciate any idea of ​​a possible solution to this.

Answer 1

Thank you for your help. I finally found out what caused the problem. In the "for" loop, later in the script, I saved the other data in a folder that I called temp. After renaming this folder to another (for example, temporary), the problem of loading data disappeared.

(Now it doesn’t matter that the practical problem is solved, but I can’t say that I understand why this specific connection arose between the later save call and this importdata or load call.)

See the new temp folder question

Answer 2

I believe your datasets are in the same directory (network or local) and with the same attributes, for example. access properties, etc.

Then there is only one option with the characteristics of the varems that are stored in these mats. Can you check how these variables are displayed in size, for example. by loading a sample. This will cut to solve your problem.

Hope this helps.

FS