Boost python multiple modules in one shared object

Question

I am trying to create a package through boost python that will contain several modules.

We believe that we want to expose a very large API, and it makes sense to group it in different modules for ease of use and to save python memory usage. On the other hand, we are compelled (for reasons beyond the scope of this question, to compile this into one common object )

So, I create with boost python a package that exports several modules, as shown below:

void exportClass1() { namespace bp = boost::python; // map the IO namespace to a sub-module // make "from myPackage.class1 import <whatever>" work bp::object class1Module(bp::handle<>(bp::borrowed(PyImport_AddModule("myPackage.class1")))); // make "from mypackage import class1" work bp::scope().attr("class1") = class1Module; // set the current scope to the new sub-module bp::scope io_scope = class1Module; // export stuff in the class1 namespace class_<class1 >("class1", init<>()) . . CLASS SPECIFICS GO HERE . Other class of module class1 go here as well } BOOST_PYTHON_MODULE(myPackage) { namespace bp = boost::python; // specify that this module is actually a package bp::object package = bp::scope(); package.attr("__path__") = "myPackage"; exportClass1(); exportClass2(); . . . } 

This code works.

The main problem is memory consumption. The general open api is very large , so downloading the entire package consumes approximately 65 MB of memory, only for all ads. (before the package user started doing anything)

This, of course, is unacceptable. (considering that loading one module should consume, possibly, 1-3 MB of RAM)

When in python, if I call:

 from myPackage.myModule import * 

OR

 from myPackage.myModule import someClass 

Memory consumption is gradually increasing to 65 MB.

After doing any import, if I call: sys.modules I see that all the classes in my package are "known", However, if I run:

 from myPackage.myModule import class1 c = class2() 

I get an error message:

NameError: name 'class2' undefined

So it seems that I am getting the worst of the two worlds, on the one hand, I consume memory, as if I imported everything from my package, on the other hand, I do not get the classes actually imported.

Any ideas how to solve this problem, so when I import only a specific module, it will be imported and not all package data will be read in python memory. (which takes time and consumes a lot of valuable memory)

Answer 1

So it was much easier than I expected.

The above code is also suitable for making calls in the form:

 from myPackage.myModule import class1 c = class2() 

What prevented the correct execution of this action are the system paths. The shared object was not placed at the python path location and there was no __init__.py in the folder where it was placed.

Once the shared object has been placed in the correct folder of site packages, which of course has __init__.py , the above example works correctly.