Numpy C API: Link Multiple Object Files

Question

I am using the numpy C API to write some functions to calculate a matrix. Today I wanted to move some parts of my functions to a separate .c file and use the header to declare them. Now I have a strange problem related to the numpy import_array function. I tried to simplify the problem as much as possible. First there is a work program:

mytest.c

 #include "mytest.h" PyObject* my_sub_function() { npy_intp dims[2] = {2, 2}; double data[] = {0.1, 0.2, 0.3, 0.4}; PyArrayObject* matrix = (PyArrayObject*)PyArray_SimpleNew(2, dims, NPY_FLOAT64); memcpy(PyArray_DATA(matrix), data, sizeof(double) * dims[0] * dims[1]); return (PyObject*)matrix; } static PyObject* my_test_function(PyObject* self, PyObject* args) { return my_sub_function(); } static PyMethodDef methods[] = { {"my_test_function", my_test_function, METH_VARARGS, ""}, {0, 0, 0, 0} }; static struct PyModuleDef module = { PyModuleDef_HEAD_INIT, "mytest", 0, -1, methods }; PyMODINIT_FUNC PyInit_mytest() { import_array(); return PyModule_Create(&module); } 

mytest.h

 #ifndef mytest_h #define mytest_h #include <Python.h> #include <numpy/arrayobject.h> #include <numpy/npy_common.h> PyObject* my_sub_function(); #endif 

Makefile

 all: mytest.o sub.o gcc -shared -Wl,-soname,mytest.so -o mytest.so mytest.o mytest.o: sub.o gcc -fPIC -c mytest.c `pkg-config --cflags python3` clean: rm -rf *.so rm -rf *.o 

Everything works as expected. I can call make and then load the module and call the function:

test.py

 import mytest print(mytest.my_test_function()) 

If I removed import_array from the init function, there will be segfault, this is the behavior that has been reported in many mailing lists and forums.

Now I just want to remove the entire my_sub_function function from mytest.c and move it to a file called sub.c :

 #include "mytest.h" PyObject* my_sub_function() { npy_intp dims[2] = {2, 2}; double data[] = {0.1, 0.2, 0.3, 0.4}; PyArrayObject* matrix = (PyArrayObject*)PyArray_SimpleNew(2, dims, NPY_FLOAT64); memcpy(PyArray_DATA(matrix), data, sizeof(double) * dims[0] * dims[1]); return (PyObject*)matrix; } 

New Makefile :

 all: mytest.o sub.o gcc -shared -Wl,-soname,mytest.so -o mytest.so mytest.o sub.o mytest.o: gcc -fPIC -c mytest.c `pkg-config --cflags python3` sub.o: gcc -fPIC -c sub.c `pkg-config --cflags python3` clean: rm -rf *.so rm -rf *.o 

If I try to load the module and call the function, the function call gives me segfault. I can solve the problem if I put the import_array call at the beginning of my_sub_function , but I don't think this is a way to use this function.

So, I would like to know why this happens and what is a “clean” way to split a numpy module into multiple source files.

Answer 1

By default, the import_array procedure will only do the NumPy C API in a single file. This is because it works through a table of function pointers stored in a static global variable (i.e. not exported and visible in only one file).

As mentioned in the documentation , you can change this behavior with several preprocessor definitions:

• In all files for your extension, define PY_ARRAY_UNIQUE_SYMBOL unique variable that is unlikely to conflict with other extensions. Including the name of your extension module in the variable name would be a good idea.

• In every file except those where you call import_array , define the symbol NO_IMPORT_ARRAY

These characters must be defined before you enable arrayobject.h for them to take effect.