Distributing Cython-based extensions using LAPACK

Question

I am writing a Python module that includes Cython extensions and uses LAPACK (and BLAS ). I am open to using clapack or lapacke or some f2c or f2py solution if necessary. The important thing is that I can call LAPACK and BLAS routines from Cython in narrow loops without Python overheads.

I found one example here . However, this example is dependent on SAGE. I want my module to be installed without installing SAGE, as my users are unlikely to want or need SAGE for anything else. My users are likely to have packages like numpy, scipy, pandas and scikit learn, so they will be reasonable dependencies. What is the optimal combination of interfaces to use, and what would the minimal setup.py file look like, could you get the necessary information (from numpy, scipy, etc.) to compile?

EDIT: This is what I ended up doing. It works on my macbook, but I have no idea how portable it is. Of course, the best way.

 from distutils.core import setup from distutils.extension import Extension from Cython.Distutils import build_ext import numpy from Cython.Build import cythonize from numpy.distutils.system_info import get_info # TODO: This cannot be the right way blas_include = get_info('blas_opt')['extra_compile_args'][1][2:] includes = [blas_include,numpy.get_include()] setup( cmdclass = {'build_ext': build_ext}, ext_modules = cythonize([Extension("cylapack", ["cylapack.pyx"], include_dirs = includes, libraries=['blas','lapack']) ]) ) 

This works because on my macbook, the clapack.h header file is in the same directory as cblas.h . I can do this in my pyx file:

 ctypedef np.int32_t integer cdef extern from "cblas.h": double cblas_dnrm2(int N,double *X, int incX) cdef extern from "clapack.h": integer dgelsy_(integer *m, integer *n, integer *nrhs, double *a, integer *lda, double *b, integer *ldb, integer * jpvt, double *rcond, integer *rank, double *work, integer * lwork, integer *info) 

Answer 1

If I understand the question correctly, you can use SciPy Cython wrappers for BLAS and LAPACK procedures. These wrappers are described here:

• BLAS
• Lapack

As stated in the documentation, you are responsible for verifying that any arrays you pass to these functions are correctly aligned for Fortran routines. You can simply import and use these functions as needed in your .pyx file. For example:

 from scipy.linalg.cython_blas cimport dnrm2 from scipy.linalg.cython_lapack cimport dgelsy 

Given that this is well-tested, widely used code that runs on different platforms, I would say that it is a good candidate for reliably distributing Cython extensions that directly call BLAS and LAPACK procedures.

---

If you do not want your code to depend on all of SciPy, you can find many relevant files for these wrapper functions in the SciPy linalg here . A useful link is these setup.py lines that list the source and header files. Note that a Fortran compiler is required!

In theory, it should be possible to isolate only the source files here, which are necessary to compile the BLAS and LAPACK Cython covers, and then link them as an independent extension to your module.

In practice, it is very difficult. The build process for the linalg submodule requires some Python functions to help compile on different platforms (for example, from here ). The building also relies on other C and Fortran source files ( here ), whose paths are hard-coded into these Python functions.

Obviously, most of the work is dedicated to ensuring that SciPy compiles correctly across different operating systems and architectures.

I'm sure this can be done, but after shuffling the files and dragging the paths, I have not yet found the right path to create this part of the linalg submodule, regardless of the rest of SciPy. If I find the right path, I will definitely confirm this answer.