How to wrap a pointer and a C length in a new-style buffer object in Cython?

Question

I am writing a Python 2.7 extension module in Cython. How to create a Python object that implements a new-style buffer interface that wraps a piece of memory provided to me by the C library? . A memory block is just a string of bytes, not a structure or a multidimensional array. I am given a pointer and the length of const void * , as well as some information about how long the pointer remains valid.

I cannot copy memory that would kill performance for my application.

With old-style buffer objects, I could just use PyBuffer_FromMemory() , but I cannot find a similarly simple way to create a new-style buffer object.

Should I create my own class that implements the buffer interface? Or does Cython provide an easy way to do this?

I read Unicode and Passing Strings and Typed memory pages from the Cython documentation, but the documentation is inaccurate and not very complete, and there are no examples similar to what I want to do.

Here is what I tried ( test.pyx ):

 from libc.stdlib cimport malloc from libc.string cimport memcpy ## pretend that this function is in some C library and that it does ## something interesting. (this function is unrelated to the problem ## I'm experiencing -- this is just an example function that returns a ## chunk of memory that I want to wrap in an object that follows the ## new buffer protocol.) cdef void dummy_function(const void **p, size_t *l): cdef void *tmp = malloc(17) memcpy(tmp, "some test\0 bytes", 17) p[0] = tmp l[0] = 17 cpdef getbuf(): cdef const void *cstr cdef size_t l dummy_function(&cstr, &l) ## error: test.pyx:21:20: Invalid base type for memoryview slice: void #cdef const void[:] ret = cstr[:l] ## error: test.pyx:24:9: Assignment to const 'ret' #cdef const char[:] ret = cstr[:l] ## error: test.pyx:27:27: Cannot convert 'void const *' to memoryviewslice #cdef char[:] ret = cstr[:l] ## this next attempt cythonizes, but raises an exception: ## $ python -c 'import test; test.getbuf()' ## Traceback (most recent call last): ## File "<string>", line 1, in <module> ## File "test.pyx", line 15, in test.getbuf (test.c:1411) ## File "test.pyx", line 38, in test.getbuf (test.c:1350) ## File "stringsource", line 614, in View.MemoryView.memoryview_cwrapper (test.c:6763) ## File "stringsource", line 321, in View.MemoryView.memoryview.__cinit__ (test.c:3309) ## BufferError: Object is not writable. cdef char[:] ret = (<const char *>cstr)[:l] ## this raises the same exception as above #cdef char[:] ret = (<char *>cstr)[:l] return ret 

Answer 1

You can determine the type of extension that implements the buffer protocol by defining __getbuffer__ and __releasebuffer__ special methods . For example:

 from cpython.buffer cimport PyBuffer_FillInfo from libc.stdlib cimport free, malloc from libc.string cimport memcpy cdef void dummy_function(const void **p, size_t *l): cdef void *tmp = malloc(17) memcpy(tmp, "some test\0 bytes", 17) p[0] = tmp l[0] = 17 cdef void free_dummy_data(const void *p, size_t l, void *arg): free(<void *>p) cpdef getbuf(): cdef const void *p cdef size_t l dummy_function(&p, &l) return MemBuf_init(p, l, &free_dummy_data, NULL) ctypedef void dealloc_callback(const void *p, size_t l, void *arg) cdef class MemBuf: cdef const void *p cdef size_t l cdef dealloc_callback *dealloc_cb_p cdef void *dealloc_cb_arg def __getbuffer__(self, Py_buffer *view, int flags): PyBuffer_FillInfo(view, self, <void *>self.p, self.l, 1, flags) def __releasebuffer__(self, Py_buffer *view): pass def __dealloc__(self): if self.dealloc_cb_p != NULL: self.dealloc_cb_p(self.p, self.l, self.dealloc_cb_arg) # Call this instead of constructing a MemBuf directly. The __cinit__ # and __init__ methods can only take Python objects, so the real # constructor is here. See: # https://mail.python.org/pipermail/cython-devel/2012-June/002734.html cdef MemBuf MemBuf_init(const void *p, size_t l, dealloc_callback *dealloc_cb_p, void *dealloc_cb_arg): cdef MemBuf ret = MemBuf() ret.p = p ret.l = l ret.dealloc_cb_p = dealloc_cb_p ret.dealloc_cb_arg = dealloc_cb_arg return ret 

With the above (named test.pyx ) you will get the following behavior:

 $ python -c 'import test; print repr(memoryview(test.getbuf()).tobytes())' 'some test\x00 bytes\x00' 

I do not know if there is an easier way.

Answer 2

Python 3.3 has the PyMemoryView_FromMemory C-API function, which creates a Python memoryview object from the supplied C buffer. memoryview objects really implement a new-style buffer interface.

If you look at its sources , you will notice that they are quite simple. It does the same thing as PyMemoryView_FromBuffer , except the first one fills Py_buffer itself.

Since the latter exists in Python 2.7, why can't we just call PyBuffer_FillInfo ourselves?

 from libc.stdlib cimport malloc from libc.string cimport memcpy cdef extern from "Python.h": ctypedef struct PyObject object PyMemoryView_FromBuffer(Py_buffer *view) int PyBuffer_FillInfo(Py_buffer *view, PyObject *obj, void *buf, Py_ssize_t len, int readonly, int infoflags) enum: PyBUF_FULL_RO cdef void dummy_function(const void **p, size_t *l): cdef void *tmp = malloc(17) memcpy(tmp, "some test\0 bytes", 17) p[0] = tmp l[0] = 17 cpdef getbuf(): cdef const void *cstr cdef size_t l cdef Py_buffer buf_info cdef char[:] ret cdef int readonly dummy_function(&cstr, &l) readonly = 1 PyBuffer_FillInfo(&buf_info, NULL, <void*>cstr, l, readonly, PyBUF_FULL_RO) ret = PyMemoryView_FromBuffer(&buf_info) return ret 

Note that, however, that the return value will have a view that looks like this: <MemoryView of 'memoryview' at 0x7f216fc70ad0> . This is because Cython seems to memoryview inside _memoryviewslice . Since memoryview objects already implement a buffer interface, perhaps you should just return the result of calling PyMemoryView_FromBuffer .

In addition, you are responsible for managing the lifetime of your buffer. memoryview objects created in this way will not automatically free memory. You must do this yourself, ensuring that you do this only after the memorybuffer refers to it. In this regard, Richard Hansen's answer is a much better alternative.