Boost.python doesn't support parallelism?

Question

I am trying to wrap a piece of C ++ code in python lib using boost.python, however I found that multiple instances cannot work at the same time:

code (C ++):

class Foo{ public: Foo(){} void run(){ int seconds = 2; clock_t endwait; endwait = clock () + seconds * CLOCKS_PER_SEC ; while (clock() < endwait) {} } }; BOOST_PYTHON_MODULE(run_test) { using namespace boost::python; class_<Foo>("test", init<>()) .def("run", &Foo::run) ; } 

which compiles using CMake (CMake):

 add_library(run_test SHARED run_test.cpp) target_link_libraries(run_test boost_python python2.7) 

and tested using the following code (Python):

 class Dos(threading.Thread): def run(self): printl('performing DoS attack') proc = test() proc.run() for i in range(5): t = Dos() t.start() 

The result shows that the code is parallelized in a very strange way. Each thread should take only 2 seconds, and four threads should run simultaneously on my quad-core computer:

 [2011-11-04 13:57:01] performing DoS attack [2011-11-04 13:57:01] performing DoS attack [2011-11-04 13:57:05] performing DoS attack [2011-11-04 13:57:05] performing DoS attack [2011-11-04 13:57:09] performing DoS attack 

Thank you for your help!

Answer 1

What you work on is the python global transition blocker. GIL allows only one thread to run in the python interpreter.

One of the benefits of Boost.Python is that you can release the GIL, make C ++, and then return it when you're done. It is also responsible. Python typically releases the GIL at regular intervals to give other threads the ability to run. If you are in C ++, this is your job. If you continue to crunch for 2 hours while holding the GIL, you will freeze the entire interpreter.

This can easily be fixed with a little inverse RAII:

 class releaseGIL{ public: inline releaseGIL(){ save_state = PyEval_SaveThread(); } inline ~releaseGIL(){ PyEval_RestoreThread(save_state); } private: PyThreadState *save_state; }; 

Now you can change your code like this:

 class Foo{ public: Foo(){} void run(){ { releaseGIL unlock = releaseGIL(); int seconds = 2; clock_t endwait; endwait = clock () + seconds * CLOCKS_PER_SEC ; while (clock() < endwait) {} } } }; 

It is VERY important to note that you SHOULD NOT touch any python code or python data or call the interpreter without holding the GIL. This will crash your interpreter.

You can also go the other way. A thread currently not containing a GIL can get it and make calls in python. It can be a thread that previously released GIL, or one that started in C ++ and never had a GIL. There is a class RAII for this:

 class AcquireGIL { public: inline AcquireGIL(){ state = PyGILState_Ensure(); } inline ~AcquireGIL(){ PyGILState_Release(state); } private: PyGILState_STATE state; }; 

Use remains as an exercise for the student.

Additional note (I always forget to mention this):

If you are going to communicate with the GIL in C ++, your module definition should start with this code:

 BOOST_PYTHON_MODULE(ModuleName) { PyEval_InitThreads(); ... }