Dynamically loading uncompiled python plugins into py2exe compilation code

Question

My Python application is built in such a way that some functions are available as plugins. The plugin architecture is currently very simple: I have a folder / plugin package that contains some python modules. I download the corresponding plugin as follows:

plugin_name = blablabla try: module = __import__(plugin_name, fromlist='do_something') except ImportError: #some error handling ... 

and then do:

 try: loans = module.do_something(id_t, pin_t) except xxx: # error handling 

I will compile the application into a Windows binary using py2exe. This works great, except for the fact that all plugins (and should be) are included in the binary. This is not very practical, because for each new plugin I have to recompile and release a new version of my application. It would be better if a new plug-in (i.e. a python file) could be copied to some folder of the application plug-in and that the Python code in the file code would be interpreted on the fly by my application.

What is the best way to do this?

(although I read every line of the selected plugin file and applied the exec statement to it. be the best ways ...)

Answer 1

If you don't mind that the plugin will be released as .py files, you can do something like the following. Put your entire plugin under the "plugin" subdirectory and create an empty "__init__.py". By executing runtime, it will import the package along with all the modules in this directory. Check out Dive In Python for an explanation ... but here is what I finally use.

 def load_plugin(path): import imp """Load plugin from directory and return list of modules""" files = os.listdir( path ) test = re.compile(".py$", re.IGNORECASE) files = filter(test.search, files) filenameToModuleName = lambda f: os.path.splitext(f)[0] moduleNames = sorted(map(filenameToModuleName, files)) f, filename, desc = imp.find_module('plugin') plugin = imp.load_module('plugin', f, filename, desc) modules = [] #print moduleNames for m in moduleNames: # skip any files starting with '__', such as __init__.py if m.startswith('__'): continue try: f, filename, desc = imp.find_module(m, plugin.__path__) modules.append( imp.load_module(m, f, filename, desc)) except ImportError: continue return modules 

Answer 2

PyInstaller allows you to import external files. If you run it on top of your application, it will not pack these files into an executable file. Then you will need to make sure that the paths are correct (that is, your application can find the modules on disk in the correct directory), and everything should work.

Answer 3

I suggest you use the pkg_resources entry_points functions (from setuptools / distribute) to implement the detection and creation of the plugin: firstly, this is the standard way to do this; secondly, it does not suffer from the problem that you mention AFAIK. All you need to do to expand the application is to pack some plugins in an egg that declares some entry points (an egg can declare many plugins), and when you install this egg in your python distribution, all the plugins that it declares, can automatically opened by your application. You can also pack your application and factory plugins into the same egg, which is quite convenient.

Answer 4

I'm not sure if you should put the plugin files in a zip library. Perhaps this is due to the fact that you are using py2exe by default when packaging your script.

You can try using compressed = False (as described in py2exe ListOfOptions ), which will eliminate the library.zip file generated by py2exe and possibly allow you to access python modules (your plugins are python modules, I suppose, from import ) in normal mode, instead of forcing them to pack into your zip or binary.

Answer 5

I found how to import external modules (on top of a compiled executable, at runtime) using pyinstaller. it shows that initially the path of the executable was automatically added to sys.path, but for security reasons they deleted it at some point. to enable it again, use:

 sys.path.append(os.path.dirname(sys.executable)) 

this will allow you to import .py files that are in the same path as the executable. you can add this line at run time or in the main application.