Passing a bytearray from Python to C and returning it

Question

I need fast XOR processing bytearray, In Python version

for i in range(len(str1)): str1[i]=str1[i] ^ 55 

very slow
I wrote this module in C. I know C very poorly before I wrote anything on it.
In option

 PyArg_ParseTuple (args, "s", &str)) 

everything works as expected, but I need to use s * instead, because the elements can contain null values, but if I change s to s * when I call python crashes

 PyArg_ParseTuple (args, "s*", &str)) // crash 

Maybe some newcomer like me wants to use my example to start writing something of my own, so bring all the information that will be used in this example to Windows.
Parsing arguments and plotting values ​​at http://docs.python.org/dev/c-api/arg.html

test_xor.c

 #include <Python.h> static PyObject* fast_xor(PyObject* self, PyObject* args) { const char* str ; int i; if (!PyArg_ParseTuple(args, "s", &str)) return NULL; for(i=0;i<sizeof(str);i++) {str[i]^=55;}; return Py_BuildValue("s", str); } static PyMethodDef fastxorMethods[] = { {"fast_xor", fast_xor, METH_VARARGS, "fast_xor desc"}, {NULL, NULL, 0, NULL} }; PyMODINIT_FUNC initfastxor(void) { (void) Py_InitModule("fastxor", fastxorMethods); } 

test_xor.py

 import fastxor a=fastxor.fast_xor("World") # it works with s instead s* print a a=fastxor.fast_xor("Wo\0rld") # It does not work with s instead s* 

compile.bat

 rem use http://bellard.org/tcc/ tiny_impdef.exe C:\Python26\python26.dll tcc -shared test_xor.c python26.def -IC:\Python26\include -LC:\Python26\libs -ofastxor.pyd test_xor.py 

Answer 1

You do not need to create an extension module to do this quickly, you can use NumPy. But for your question, you'll need c code like this:

 #include <Python.h> #include <stdlib.h> static PyObject * fast_xor(PyObject* self, PyObject* args) { const char* str; char * buf; Py_ssize_t count; PyObject * result; int i; if (!PyArg_ParseTuple(args, "s#", &str, &count)) { return NULL; } buf = (char *)malloc(count); for(i=0;i<count;i++) { buf[i]=str[i] ^ 55; } result = Py_BuildValue("s#", buf, count); free(buf); return result; } 

You cannot change the contents of a string object because a string in Python is immutable. You can use "s #" to get the char * pointer and the length of the buffer.

If you can use NumPy:

 In [1]: import fastxor In [2]: a = "abcdsafasf12q423\0sdfasdf" In [3]: fastxor.fast_xor(a) Out[3]: 'VUTSDVQVDQ\x06\x05F\x03\x05\x047DSQVDSQ' In [5]: import numpy as np In [6]: (np.frombuffer(a, np.int8)^55).tostring() Out[6]: 'VUTSDVQVDQ\x06\x05F\x03\x05\x047DSQVDSQ' In [7]: a = a*10000 In [8]: %timeit fastxor.fast_xor(a) 1000 loops, best of 3: 877 us per loop In [15]: %timeit (np.frombuffer(a, np.int8)^55).tostring() 1000 loops, best of 3: 1.15 ms per loop 

Answer 2

This can be done very quickly using numpy. You are unlikely to get a faster manual transfer of your own xor procedure to C:

 In [1]: import numpy In [2]: data = numpy.uint8(numpy.random.randint(0, 256, 10000)) In [3]: timeit xor_data = numpy.bitwise_xor(data, 55) 100000 loops, best of 3: 17.4 us per loop 

If you use a large data set (say, 100 million points), this compares favorably with the time you specified for your code:

 In [12]: data = numpy.uint8(numpy.random.randint(0, 256, 100000000)) In [13]: timeit xor_data = numpy.bitwise_xor(data, 55) 1 loops, best of 3: 198 ms per loop 

Answer 3

An alternative approach is to use PyObject_GetBuffer . The module below defines fast_xor for any object that supports the buffer protocol, and fast_xor_inplace for objects with rewritable buffers, such as bytearray . This version returns None . I also added a second unsigned char argument with a default value of 55.

Example:

 >>> s = 'abc' >>> b = bytearray(s) >>> fast_xor(s), fast_xor(s, 0x20) ('VUT', 'ABC') >>> fast_xor_inplace(b, 0x20) >>> b bytearray(b'ABC') >>> fast_xor_inplace(s) Traceback (most recent call last): File "<stdin>", line 1, in <module> BufferError: Object is not writable. >>> fast_xor(b, 256) Traceback (most recent call last): File "<stdin>", line 1, in <module> OverflowError: unsigned byte integer is greater than maximum 

A source:

 #include <Python.h> static PyObject *fast_xor_inplace(PyObject *self, PyObject *args) { PyObject *arg1; unsigned char arg2 = 55; Py_buffer buffer; char *buf; int i; if (!PyArg_ParseTuple(args, "O|b:fast_xor_inplace", &arg1, &arg2)) return NULL; if (PyObject_GetBuffer(arg1, &buffer, PyBUF_WRITABLE) < 0) return NULL; buf = buffer.buf; for(i=0; i < buffer.len; i++) buf[i] ^= arg2; PyBuffer_Release(&buffer); Py_INCREF(Py_None); return Py_None; } static PyObject *fast_xor(PyObject *self, PyObject *args) { PyObject *arg1; unsigned char arg2 = 55; PyObject *result; Py_buffer buffer; char *buf, *str; int i; if (!PyArg_ParseTuple(args, "O|b:fast_xor", &arg1, &arg2)) return NULL; if (PyObject_GetBuffer(arg1, &buffer, PyBUF_SIMPLE) < 0) return NULL; result = PyString_FromStringAndSize(NULL, buffer.len); if (result == NULL) return NULL; buf = buffer.buf; str = PyString_AS_STRING(result); for(i=0; i < buffer.len; i++) str[i] = buf[i] ^ arg2; PyBuffer_Release(&buffer); return result; } static PyMethodDef fastxorMethods[] = { {"fast_xor", fast_xor, METH_VARARGS, "fast xor"}, {"fast_xor_inplace", fast_xor_inplace, METH_VARARGS, "fast inplace xor"}, {NULL, NULL, 0, NULL} }; PyMODINIT_FUNC initfastxor(void) { Py_InitModule3("fastxor", fastxorMethods, "fast xor functions"); } 

Answer 4

Try

  char pybuffer[1024]; ... PyArg_ParseTuple (args, "s#", pybuffer, sizeof(pybuffer)); 

I don’t know what the buffer size requirements are. I suggested that 1024 is adequate.

Answer 5

Do not use for -loop. Use an understanding list , they are much faster:

 In [1]: import random In [2]: t = bytearray([random.randint(0,255) for i in xrange(10000)]) In [3]: u = bytearray([b^55 for b in t]) 

It's very fast:

 In [11]: %timeit u = bytearray([b^55 for b in t]) 1000 loops, best of 3: 1.36 ms per loop 

It is not very slow. For 1 MB (10 ** 6 bytes), this takes about 130 ms.

Of course, using numpy, as Henry Homersall answered, is the best solution.