How to handle C ++ std :: vector <int> return type in Python types?

Question

I cannot find how ctypes will bridge the gap between std::vector and Python; no, where the mentioned combination is mentioned on the Internet. Is this bad practice, does it not exist or am I missing something?

C ++ : xxx.cpp

 #include <fstream> #include <string> using namespace std; extern "C" std::vector<int> foo(const char* FILE_NAME) { string line; std::vector<int> result; ifstream myfile(FILE_NAME); while (getline(myfile, line)) { result.push_back(1); } return(result); } 

Python: xxx.py

 import ctypes xxx = ctypes.CDLL("./libxxx.so") xxx.foo.argtypes = ?????? xxx.foo.restype = ?????? 

Answer 1

A special reason is that speed is important. I am creating an application that should be able to process big data. For 200,000 rows, gaps should be calculated for 300 values ​​(200 thousand for 300 matrices). I believe, but correct me if I am mistaken that C ++ will be much faster.

Well, if you are reading from a large file, your process will be mainly related to IO, so the timings between Python and C will probably not differ significantly.

The following code ...

 result = [] for line in open('test.txt'): result.append(line.count('NA')) 

... seems to work as fast as anything I can hack in C, although it uses some optimized algorithm I'm not really familiar with.

It takes less than a second to process 200,000 lines, although I would be interested to know if it is possible to write a C function that is much faster.

---

Update

If you want to do this in C and end up with a Python list, it's probably more efficient to use the Python / C API to create a list yourself, rather than creating a std::vector , and then convert to a Python list later.

An example that simply returns a list of integers from 0 to 99 ...

 // hack.c #include <python2.7/Python.h> PyObject* foo(const char* filename) { PyObject* result = PyList_New(0); int i; for (i = 0; i < 100; ++i) { PyList_Append(result, PyInt_FromLong(i)); } return result; } 

Compiled with ...

 $ gcc -c hack.c -fPIC $ ld -o hack.so -shared hack.o -lpython2.7 

Usage example ...

 >>> from ctypes import * >>> dll = CDLL('./hack.so') >>> dll.foo.restype = py_object >>> dll.foo('foo') [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, ...] 

Answer 2

Regardless of whether this approach provides faster runtimes, I will talk a bit about how you can do this. Basically, create a pointer to a C ++ vector that can interact with Python through C functions. Then you can wrap the C ++ code in a Python class, hiding the implementation details of ctypes .

I have included what I thought would be a useful magic method for inclusion in a Python class. You can remove them or add more according to your needs. Destructor is very important to keep.

C ++

 // vector_python.cpp #include <vector> #include <iostream> #include <fstream> #include <string> using namespace std; extern "C" void foo(vector<int>* v, const char* FILE_NAME){ string line; ifstream myfile(FILE_NAME); while (getline(myfile, line)) v->push_back(1); } extern "C" { vector<int>* new_vector(){ return new vector<int>; } void delete_vector(vector<int>* v){ cout << "destructor called in C++ for " << v << endl; delete v; } int vector_size(vector<int>* v){ return v->size(); } int vector_get(vector<int>* v, int i){ return v->at(i); } void vector_push_back(vector<int>* v, int i){ v->push_back(i); } } 

Compile it as a shared library. On Mac OS X, it might look like this:

 g++ -c -fPIC vector_python.cpp -o vector_python.o g++ -shared -Wl,-install_name,vector_python_lib.so -o vector_python_lib.so vector_python.o 

Python

 from ctypes import * class Vector(object): lib = cdll.LoadLibrary('vector_python_lib.so') # class level loading lib lib.new_vector.restype = c_void_p lib.new_vector.argtypes = [] lib.delete_vector.restype = None lib.delete_vector.argtypes = [c_void_p] lib.vector_size.restype = c_int lib.vector_size.argtypes = [c_void_p] lib.vector_get.restype = c_int lib.vector_get.argtypes = [c_void_p, c_int] lib.vector_push_back.restype = None lib.vector_push_back.argtypes = [c_void_p, c_int] lib.foo.restype = None lib.foo.argtypes = [c_void_p] def __init__(self): self.vector = Vector.lib.new_vector() # pointer to new vector def __del__(self): # when reference count hits 0 in Python, Vector.lib.delete_vector(self.vector) # call C++ vector destructor def __len__(self): return Vector.lib.vector_size(self.vector) def __getitem__(self, i): # access elements in vector at index if 0 <= i < len(self): return Vector.lib.vector_get(self.vector, c_int(i)) raise IndexError('Vector index out of range') def __repr__(self): return '[{}]'.format(', '.join(str(self[i]) for i in range(len(self)))) def push(self, i): # push calls vector push_back Vector.lib.vector_push_back(self.vector, c_int(i)) def foo(self, filename): # foo in Python calls foo in C++ Vector.lib.foo(self.vector, c_char_p(filename)) 

Then you can check it in the interpreter ( file.txt consists of only three jibberish lines).

 >>> from vector import Vector >>> a = Vector() >>> a.push(22) >>> a.push(88) >>> a [22, 88] >>> a[1] 88 >>> a[2] Traceback (most recent call last): File "<stdin>", line 1, in <module> File "vector.py", line 30, in __getitem__ raise IndexError('Vector index out of range') IndexError: Vector index out of range >>> a.foo('file.txt') >>> a [22, 88, 1, 1, 1] >>> b = Vector() >>> ^D destructor called in C++ for 0x1003884d0 destructor called in C++ for 0x10039df10 

Answer 3

In principle, returning a C ++ object from a dynamically loaded library is not a good idea. To use C ++ vector in Python code, you must teach Python to deal with C ++ objects (and this includes a binary representation of objects that can be modified using the new version of the C ++ compiler or STL).

ctypes allows you to interact with the library using C. types. Not C ++.

Perhaps the problem is solvable with boost::python , but it is more reliable to use equal C for interaction.