Python: is iterating a multidimensional array super slow?

Question

I need to iterate all the elements in a two-dimensional array of integers and change the value (according to some rule, it doesn't matter).

I am surprised at how significant the performance difference is between python runtime and C # or java runtime. I wrote completely wrong python code (v2.7.2)?

import numpy a = numpy.ndarray((5000,5000), dtype = numpy.int32) for x in numpy.nditer(aT): x = 123 

 >python -m timeit -n 2 -r 2 -s "import numpy; a = numpy.ndarray((5000,5000), dtype=numpy.int32)" "for x in numpy.nditer(aT):" " x = 123" 2 loops, best of 2: 4.34 sec per loop 

For example, C # code only performs 50 ms, i.e. python is almost 100 times slower ! (suppose the matrix variable is already initialized)

 for (y = 0; y < 5000; y++) for (x = 0; x < 5000; x++) matrix[y][x] = 123; 

Answer 1

Python is a much more dynamic language than C or C #. The main reason the loop is so slow is that on each pass the CPython interpreter does an extra job that takes time: in particular, it connects the name x to the next object from the iterator, and then when it evaluates it needs to find it again name x .

As @Sven Marnach noted, you can call the function of the numpy.fill() method, and that is fast. This function is compiled by C, or perhaps Fortran, and it will simply numpy.array over the addresses of the numpy.array data numpy.array and fill in the values. Much less dynamic than Python, which is good for this simple case.

But now consider PyPy. After starting your program under PyPy, the JIT analyzes what your code actually does. In this example, he notes that the name x not used for anything other than the destination, and can optimize the name binding. This example should be what PyPy greatly speeds up; PyPy will probably be ten times faster than regular Python (so only one tenth faster than C, not 1/100 faster).

http://pypy.org

As I understand it, PyPy will not work with Numpy yet, so you cannot just run existing Numpy code under PyPy. But the day is coming.

I am very happy PyPy. He offers the hope that we can write at a very high level of the language (Python), but at the same time we get almost the performance of writing things in the "assembler language" (C). For examples like this, Numpy can even surpass the performance of naive C code by using SIMD instructions from the CPU (SSE2, NEON, or something else). In this SIMD example, you can set four integers to 123 with each cycle, and it will be faster than a simple C cycle. (Unless the C compiler has used SIMD optimization! Which, think about it, most likely for this case. Therefore, we will return to “almost C speed,” rather than faster for this example, but we can imagine more complex cases where the C compiler is not smart enough to optimize where the future PyPy might be.)

But do not pay attention to PyPy. If you will be working with Numpy, it is recommended that you examine all the functions, such as numpy.fill() , that speed up your code.

Answer 2

Oops! Iterating through numpy arrays in python is slow. (Slower than iterating through a python list, too).

As a rule, you avoid iterating through them directly.

If you can give us an example of a rule based on which you change things, there is a good chance that it can be easily vectorized.

As an example of a toy:

 import numpy as np x = np.linspace(0, 8*np.pi, 100) y = np.cos(x) x[y > 0] = 100 

However, in many cases, you must iterate either because of an algorithm (for example, the finite difference method), or to reduce the cost of memory of temporary arrays.

In this case, look at Cython , Weave or something similar.

Answer 3

The example you quoted supposedly meant that all elements of the two-dimensional NumPy array are 123. This can be done like this:

 a.fill(123) 

or

 a[:] = 123 

Answer 4

C ++ emphasizes the machine time of a programmer.

Python emphasizes programmer time over machine time.

Pypy is a python written in python and it has a numpy start; you can try this. Pypy has a good JIT that does things pretty quickly.

You can also try cython, which allows you to translate a Python dialect into C and compile C into a Python C extension module; this allows you to continue to use CPython for most of your code, but it speeds up a bit. However, in one microbenchmark I tried to compare Pypy and Cython, Pypy was a little faster than Cython.

Cython uses highly python syntax, but it allows you to mix Python data types with C data types fairly freely. If you redo your hotspots using C data types, this should be pretty fast. Continuing to use Python data types also accelerates Cython, but not so much.

Answer 5

The nditer code nditer not assign values ​​to a elements. This does not affect the timing problem, but I mention it because it should not be taken as a good use of nditer .

correct version:

 for i in np.nditer(a, op_flags=[["readwrite"]]): i[...] = 123 

[...] necessary to maintain a reference to the value of the loop, which is an array of form () .

It makes no sense to use AT , since the values ​​of the base a are changing.

I agree that the correct way to accomplish this assignment is a[:]=123 .

Answer 6

If you need to do operations on a multidimensional array, which depends on the value of the array, but does not depend on the position inside the array, then .itemset is 5 times faster than nditer for me.

So instead of doing something like

 image = np.random.random_sample((200, 200,3)); with np.nditer(image, op_flags=['readwrite']) as it: for x in it: x[...] = x*4.5 if x<0.2 else x 

You can do it

 image2 = np.random.random_sample((200, 200,3)); for i in range(0,image2.size): x = image2.item(i) image2.itemset(i, x*4.5 if x<0.2 else x);