How can I recognize classes in a specific package in python?

Question

I have a plugin module module package. It looks like this:

 / Plugins 
 /Plugins/__init__.py
 /Plugins/Plugin1.py
 /Plugins/Plugin2.py 
 etc ...

Each .py file contains a class derived from PluginBaseClass . Therefore, I need to list each module in the Plugins package, and then look for any classes that implement PluginBaseClass . Ideally, I want to do something like this:

 for klass in iter_plugins(project.Plugins): action = klass() action.run() 

I saw some other answers there, but my situation is different. I have the actual import into the base package (i.e. import project.Plugins ) and I need to find the classes after the modules are detected.

Answer 1

Scanning modules are not a good idea. If you need a class registry, you should look at metaclasses or use existing solutions like zope.interface . A simple solution using metaclasses might look like this:

 from functools import reduce class DerivationRegistry(type): def __init__(cls,name,bases,cls_dict): type.__init__(cls,name,bases,cls_dict) cls._subclasses = set() for base in bases: if isinstance(base,DerivationRegistry): base._subclasses.add(cls) def getSubclasses(cls): return reduce( set.union, ( succ.getSubclasses() for succ in cls._subclasses if isinstance(succ,DerivationRegistry)), cls._subclasses) class Base(object): __metaclass__ = DerivationRegistry class Cls1(object): pass class Cls2(Base): pass class Cls3(Cls2,Cls1): pass class Cls4(Cls3): pass print(Base.getSubclasses()) 

Answer 2

You can (and probably should) define __all__ in __init__.py as a list of submodules in your package; this means that you support people doing from Plugins import * . If you have done this, you can iterate over the modules with

 import Plugins import sys modules = { } for module in Plugins.__all__: __import__( module ) modules[ module ] = sys.modules[ module ] # iterate over dir( module ) as above 

The reason the other answer posted here is failing is because __import__ imports the lower level module, but returns the upper level (see docs ). I do not know why.

Answer 3

Edit: here is a revised solution. I realized that I was making a mistake when testing my previous one, and this really is not the way you expected. So here is a more complete solution:

 import os from imp import find_module from types import ModuleType, ClassType def iter_plugins(package): """Receives package (as a string) and, for all of its contained modules, generates all classes that are subclasses of PluginBaseClass.""" # Despite the function name, "find_module" will find the package # (the "filename" part of the return value will be None, in this case) filename, path, description = find_module(package) # dir(some_package) will not list the modules within the package, # so we explicitly look for files. If you need to recursively descend # a directory tree, you can adapt this to use os.walk instead of os.listdir modules = sorted(set(i.partition('.')[0] for i in os.listdir(path) if i.endswith(('.py', '.pyc', '.pyo')) and not i.startswith('__init__.py'))) pkg = __import__(package, fromlist=modules) for m in modules: module = getattr(pkg, m) if type(module) == ModuleType: for c in dir(module): klass = getattr(module, c) if (type(klass) == ClassType and klass is not PluginBaseClass and issubclass(klass, PluginBaseClass)): yield klass 

My previous solution was:

You can try something like:

 from types import ModuleType import Plugins classes = [] for item in dir(Plugins): module = getattr(Plugins, item) # Get all (and only) modules in Plugins if type(module) == ModuleType: for c in dir(module): klass = getattr(module, c) if isinstance(klass, PluginBaseClass): classes.append(klass) 

Actually, even better if you want modularity:

 from types import ModuleType def iter_plugins(package): # This assumes "package" is a package name. # If it the package itself, you can remove this __import__ pkg = __import__(package) for item in dir(pkg): module = getattr(pkg, item) if type(module) == ModuleType: for c in dir(module): klass = getattr(module, c) if issubclass(klass, PluginBaseClass): yield klass 

Answer 4

If you don’t know what will happen in Plugins well in advance, you can get a list of python files in the package directory and import them like this:

 # compute a list of modules in the Plugins package import os import Plugins plugin_modules = [f[:-3] for f in os.listdir(os.path.dirname(Plugins.__file__)) if f.endswith('.py') and f != '__init__.py'] 

Sorry, this understanding may be a sip for someone relatively new to python. Here's a more detailed version (it might be easier to follow):

 plugin_modules = [] package_path = Plugins.__file__ file_list = os.listdir(os.path.dirname(package_path)) for file_name in file_list: if file_name.endswith('.py') and file_name != '__init__.py': plugin_modules.append(file_name) 

Then you can use __import__ to get the module:

 # get the first one plugin = __import__('Plugins.' + plugin_modules[0])