Error with OMP_NUM_THREADS when using distributed package

Question

I use distributed , a structure that allows parallel computing. This is my main use case - with NumPy. When I include the NumPy code, which relies on np.linalg , I get an error message with OMP_NUM_THREADS , which is associated with the OpenMP library .

Minimal example:

 from distributed import Executor import numpy as np e = Executor('144.92.142.192:8786') def f(x, m=200, n=1000): A = np.random.randn(m, n) x = np.random.randn(n) # return np.fft.fft(x) # tested; no errors # return np.random.randn(n) # tested; no errors return A.dot(y).sum() # tested; throws error below s = [e.submit(f, x) for x in [1, 2, 3, 4]] s = e.gather(s) 

When I test the linalg test, e.gather fails, as each job raises the following error:

 OMP: Error #34: System unable to allocate necessary resources for OMP thread: OMP: System error #11: Resource temporarily unavailable OMP: Hint: Try decreasing the value of OMP_NUM_THREADS. 

What should I set OMP_NUM_THREADS to?

Answer 1

Short answer

 export OMP_NUM_THREADS=1 or dask-worker --nthreads 1 

Explanation

The OMP_NUM_THREADS environment OMP_NUM_THREADS controls the number of threads that many libraries, including the numpy.dot BLAS power numpy.dot , use, for example, a matrix in their calculations.

The conflict is that you have two parallel libraries that call each other, BLAS and dask.distributed. Each library is designed to use as many threads as possible, since the system has logical kernels.

For example, if you had eight cores, then dask.distributed can run your function f eight times in a row on different threads. A call to the numpy.dot function in f will use eight threads for each call, as a result of which a stream of 64 threads will be executed immediately.

This is actually great, you have run into a performance hit, but everything may work correctly, but it will be slower than if you only used eight threads at a time, either by restricting dask.distributed or by limiting BLAS.

Your system probably has OMP_THREAD_LIMIT installed on some reasonable amount, such as 16, to alert you to this event when this happens.