Why doesn't numpy calculation affect global interpreter lock?

I'm trying to decide whether to use multiprocessor or streaming, and I found out some interesting snippets about Global Interpreter Lock . In this nice blog post, it seems that multithreading is not suitable for busy tasks. However, I also found out that some functions, such as I / O or numpy, are not affected by GIL.

Can someone explain why, and how can I find out if my (possibly quite multi-dimensional) code is suitable for multithreading?

+6
source share
1 answer

Many numpy calculations do not affect the GIL, but not all.

In code that does not require a Python interpreter (such as C libraries), you can specifically release the GIL to continue working with other code that depends on the interpreter. In the Numpy C code base, the macros NPY_BEGIN_THREADS and NPY_END_THREADS are used to distinguish between blocks of code that allow the release of GIL. You can see them in this numpy source search .

The NumPy C API documentation contains additional information about thread support. Note the additional macros NPY_BEGIN_THREADS_DESCR , NPY_END_THREADS_DESCR and NPY_BEGIN_THREADS_THRESHOLDED , which handle conditional GIL release, depending on the dtypes array and the size of the loops.

Most core functions release the GIL - for example, Universal Functions (ufunc) do this as described :

as long as no arrays of objects are involved, Python Global Interpreter Lock (GIL) is released before calling the loops. If necessary, it is restored to handle error conditions.

As for your own code, the source code for NumPy is available . Check the functions used (and the functions that they call) for the above macros. Note also that the performance advantage depends heavily on how long the GIL has been released - if your code constantly crashes to / from Python, you will not see much of the improvement.

Another option is to just check it out. However, keep in mind that functions using conditional GIL macros can exhibit different behavior with small and large arrays. Therefore, a test with a small data set may not be an accurate representation of performance for a larger task.

There is additional information on parallel processing with accessible numpy on the official wiki and a useful Python GIL entry in general at Programmers.SE .

+12
source

All Articles