How to control Python files for changes?

Question

I want to restart my Python web application if the code changes. But there may be a large number of files that could be changed, since files in imported modules can change ...

How to get actual file names from imported packages / modules?

How can Python files be defined efficiently? Is there a library for this?

Answer 1

Shameless plug. There's also http://github.com/gorakhargosh/watchdog that I'm working on to do just that.

NTN.

Answer 2

gamin is another option that is slightly smaller than Linux.

Answer 3

I’m not sure how you would use the “reload application” operation in your circumstances; reloading the modified module with the built-in reload will probably not clip it.

But as for determining whether there was a change, it would be one way to get closer to it.

• Most python modules have the __file__ attribute.
• All loaded modules are stored in sys.modules .
• We can go through sys.modules at a certain interval and look for changes on the disk for each module in turn

Sometimes __file__ points to a .pyc file instead of a .py file, so you may have to disable the final c. Sometimes a .pyc file exists, but a .py does not exist; in a reliable system you would have to allow it.

Proof of the code concept for this (not reliable):

 _module_timestamps = {} _checking = False def run_checker(): global _checking _checking = True while _checking: for name, module in sys.modules.iteritems(): if hasattr(module, '__file__'): filename = module.__file__ if filename.endswith('.pyc'): filename = filename[:-1] mtime = os.stat(filename).st_mtime if name not in _module_timestamps: _module_timestamps[name] = mtime else: if mtime > _module_timestamps[name]: do_reload(name) else: 'module %r has no file attribute' % (name,) time.sleep(1) def do_reload(modname): print 'I would reload now, because of %r' % (modname,) check_thread = threading.Thread(target=run_checker) check_thread.daemon = True check_thread.start() try: while 1: time.sleep(0.1) except KeyboardInterrupt: print '\nexiting...' 

Answer 4

Here is an example of how this can be implemented using pyinotify (i.e. on Linux).

 from importlib import import_module class RestartingLauncher: def __init__(self, module_name, start_function, stop_function, path="."): self._module_name = module_name self._filename = '%s.py' % module_name self._start_function = start_function self._stop_function = stop_function self._path = path self._setup() def _setup(self): import pyinotify self._wm = pyinotify.WatchManager() self._notifier = pyinotify.ThreadedNotifier( self._wm, self._on_file_modified) self._notifier.start() # We monitor the directory (instead of just the file) because # otherwise inotify gets confused by editors such a Vim. flags = pyinotify.EventsCodes.OP_FLAGS['IN_MODIFY'] wdd = self._wm.add_watch(self._path, flags) def _on_file_modified(self, event): if event.name == self._filename: print "File modification detected. Restarting application..." self._reload_request = True getattr(self._module, self._stop_function)() def run(self): self._module = import_module(self._module_name) self._reload_request = True while self._reload_request: self._reload_request = False reload(self._module) getattr(self._module, self._start_function)() print 'Bye!' self._notifier.stop() def launch_app(module_name, start_func, stop_func): try: import pyinotify except ImportError: print 'Pyinotify not found. Launching app anyway...' m = import_module(self._module_name) getattr(m, start_func)() else: RestartingLauncher(module_name, start_func, stop_func).run() if __name__ == '__main__': launch_app('example', 'main', 'force_exit') 

The parameters in the launch_app call are the file name (without ".py"), a function to start execution, and a function that somehow stops the execution.

Here is a silly example of an “application” that can be overwritten using the previous code:

 run = True def main(): print 'in...' while run: pass print 'out' def force_exit(): global run run = False 

In a typical application where you want to use this, you will probably have some kind of main loop. Here's a more realistic example for a GLib / GTK + application:

 from gi.repository import GLib GLib.threads_init() loop = GLib.MainLoop() def main(): print "running..." loop.run() def force_exit(): print "stopping..." loop.quit() 

The same concept works for most other loops (Clutter, Qt, etc.).

Monitoring multiple code files (i.e. all files that are part of the application) and error tolerance (for example, printing exceptions and idle wait until the code is fixed and then run it again) will remain as exercises for the reader :).

Note. All of the code in this answer is released under the ISC license (in addition to Creative Commons).

Answer 5

This is a specific operating system. For Linux there is inotify, see, for example, http://github.com/rvoicilas/inotify-tools/