I am working on a science cluster, which was recently updated by the administrator, and now my code is superslow, while it was decent. I am using python 3.4
How it works: I have to guess what the administrator has changed, and then ask him to make the appropriate changes, because if I ask him a direct question, we won’t do anything.
So, I ran my code with the profiler, and I found that there are several routines that are called many times, these routines:
- built-in array of methods (called ~ 10 ^ 5, runtime 0.003s)
- type numpy.ndarray (~ 5000, 0.03s)
- mtrand.RandomState uniformity (~ 2000, 0.03s)
I assume that some of these libraries were parallelized in the previous installed version of python, for example, were connected to mpi-parallel or multidisciplinary libraries of the mathematical core.
I would like to know if I guess correctly, or I need to think about something else, because my code itself has not changed.
The routines I quoted here are most relevant because they account for 85% of the total time. in particular, the array takes 55% if the total time. The efficiency of my code has been reduced by 10 times. Before speaking with the system manager, I would like to receive confirmation that these procedures have a parallel version.
, , . , numpy.array 8 , , , 2 . top , (~ 0,1%), 100%.
In [3]: numpy.__config__.show()
lapack_info:
libraries = ['lapack']
library_dirs = ['/usr/lib64']
language = f77
atlas_threads_info:
libraries = ['satlas']
library_dirs = ['/usr/lib64/atlas']
define_macros = [('ATLAS_WITHOUT_LAPACK', None)]
language = c
include_dirs = ['/usr/include']
blas_opt_info:
libraries = ['satlas']
library_dirs = ['/usr/lib64/atlas']
define_macros = [('ATLAS_INFO', '"\\"3.10.1\\""')]
language = c
include_dirs = ['/usr/include']
atlas_blas_threads_info:
libraries = ['satlas']
library_dirs = ['/usr/lib64/atlas']
define_macros = [('ATLAS_INFO', '"\\"3.10.1\\""')]
language = c
include_dirs = ['/usr/include']
openblas_info:
NOT AVAILABLE
lapack_opt_info:
libraries = ['satlas', 'lapack']
library_dirs = ['/usr/lib64/atlas', '/usr/lib64']
define_macros = [('ATLAS_WITHOUT_LAPACK', None)]
language = f77
include_dirs = ['/usr/include']
lapack_mkl_info:
NOT AVAILABLE
blas_mkl_info:
NOT AVAILABLE
mkl_info:
NOT AVAILABLE
ldd /usr/lib64/python3.4/site-packages/numpy/core/_dotblas.cpython-34m.so
linux-vdso.so.1 => (0x00007fff46172000)
libsatlas.so.3 => /usr/lib64/atlas/libsatlas.so.3 (0x00007f0d941a0000)
libpython3.4m.so.1.0 => /lib64/libpython3.4m.so.1.0 (0x00007f0d93d08000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f0d93ae8000)
libc.so.6 => /lib64/libc.so.6 (0x00007f0d93728000)
libgfortran.so.3 => /lib64/libgfortran.so.3 (0x00007f0d93400000)
libm.so.6 => /lib64/libm.so.6 (0x00007f0d930f8000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f0d92ef0000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007f0d92ce8000)
/lib64/ld-linux-x86-64.so.2 (0x00007f0d950e0000)
libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007f0d92aa8000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f0d92890000)
Numpy , libpthread.so( , , ?).
, numpy 1.8.2 1.9.2, array 5 300. , , , , (, , !)