Geek Answers Handbook

How to "stop" and "resume" a long time Python script?

I wrote a Python script that processes a large number of large text files and can run a lot of time . Sometimes you need to stop the script from starting and resume it later. Possible reasons for stopping the script are a program crash, an out-of-space disk situation, or many others when you need to do this. I want to implement a stop / resume mechanism for a script.

  • In stop : the script exits and saves the current state.
  • In summary : a script is launched, but continues from the last saved state

I am going to implement it using brine and signal modules.

I will be happy to hear how to do this in a pythonic way.

Thanks!

+8
python pickle
source share
3 answers

Here is something simple that I hope can help you:

import time import pickle REGISTRY = None def main(start=0): """Do some heavy work ...""" global REGISTRY a = start while 1: time.sleep(1) a += 1 print a REGISTRY = pickle.dumps(a) if __name__ == '__main__': print "To stop the script execution type CTRL-C" while 1: start = pickle.loads(REGISTRY) if REGISTRY else 0 try: main(start=start) except KeyboardInterrupt: resume = raw_input('If you want to continue type the letter c:') if resume != 'c': break

Launch Example:

 $ python test.py To stop the script execution type CTRL-C 1 2 3 ^CIf you want to continue type the letter c:c 4 5 6 7 8 9 ^CIf you want to continue type the letter c: $ python test.py
+4
source share

If you want to read large files, just use the file descriptor and read the lines one at a time, processing each line as you need. If you want to save a python session, just use dill.dump_session - and it will save all existing objects. Other answers will fail because pickle cannot determine the writing file. dill , however, can serialize almost every python object - including a file descriptor.

 Python 2.7.9 (default, Dec 11 2014, 01:21:43) [GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import dill >>> f = open('bigfile1.dat', 'r') >>> data = f.readline() >>> >>> dill.dump_session('session.pkl') >>>

Then close the python session and restart. When you load_session , you load all the objects that existed during the dump_session call.

 dude@hilbert>$ python Python 2.7.9 (default, Dec 11 2014, 01:21:43) [GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import dill >>> dill.load_session('session.pkl') >>> len(data) 9 >>> data += f.readline() >>> f.close() >>>

Just like that.

Get the dill here: https://github.com/uqfoundation

+1
source share

Execution can save a life or (except for security exceptions), the state of the script can be pickle d, zipped and saved.

http://docs.python.org/library/pickle.html

http://docs.python.org/library/marshal.html

http://docs.python.org/library/stdtypes.html (5.9)

http://docs.python.org/library/archiving.html

http://www.henrysmac.org/?p=531

0
source share

More articles:


All Articles