Geek Answers Handbook

Function to save and load python memory?

I wrote a python program that takes several hours to calculate. Now I want the program to save all the memory from time to time (mostly numpy arrays). Thus, I can restart the calculations, starting from the moment when the last salvation occurred. I'm not looking for something like "numpy.save (file, arr)", but a way to save all the memory in one go ...

Regards, Matthias

+4
source share
2 answers

I agree with @phyrox what dillyou can use to save your live objects to disk so you can restart them later. dillcan serialize arrays numpywith dump(), and the entire interpreter session with dump_session().

However, it looks like you are really asking about some form of caching ... so I have to say that the comment from @Alfe is probably a little closer to what you want. If you want seamless caching and archiving arrays to memory ... then you want joblibor klepto.

kleptobuilt on top dilland can cache the inputs and outputs of the function in memory (so that the calculations are not performed twice), and it can easily store objects in the cache to disk or to the database.

github - , . https://github.com/uqfoundation/klepto https://github.com/joblib/joblib. Klepto , , joblib. Joblib , - .

klepto: https://github.com/uqfoundation/klepto/blob/master/tests/test_workflow.py

, numpy: https://github.com/uqfoundation/klepto/blob/master/tests/test_cache.py

+1

Dill : https://pypi.python.org/pypi/dill

Dill , "pickle", . , . , , , , , , "" .

:

import dill as pickle;
from numpy import array;

a = array([1,2]);
pickle.dump_session('sesion.pkl')
a = 0;
pickle.load_session('sesion.pkl')
print a;

"pickle", http://docs.python.org/library/pickle.html , import dill as pickle

, , . .

0

More articles:


All Articles