Geek Answers Handbook

Python limitation Numpy.ndarray.shape

I want to create a Numpy matrix in Python with the following code:

import numpy result=numpy.zeros((20,20,20,30,30,30)) numpy.save('result',result)

I get the following error:

 Traceback (most recent call last): File "numpy_memoryerror.py", line 5, in <module> result=numpy.zeros((20,20,20,30,30,30)) MemoryError

If I use smaller sizes such as:

 result=numpy.ones((10,10,10,20,20,20))

then the code works.

Can someone tell me the limit of a tuple of a form?

+4
source share
1 answer

This is not the fundamental limit of a tuple of a form; you do not have enough memory (RAM) on your system, therefore MemoryError .

Again 20 * 20 * 20 * 30 * 30 * 30 - this is 216 million 64-bit (8 bytes) floating or a little more than 1.6 GB of RAM. So, do you have 1.6 GB of RAM when running the script at this point? (Do not forget all the RAM used by python, OS, other running programs, etc.). If you are on linux / unix, you can see how much free memory by typing free -m from the command line. In the windows you can see the free memory by going to the task manager. In addition, some operating systems limit the amount of memory that a single process can allocate (for example, python); for example, 32-bit windows provide only 2 GB of address space for each process).

Compare this to 20 * 20 * 20 * 10 * 10 * 10, which is only ~ 0.06 GB (or 27 times less memory).

If you don't need 8-byte floats, you can do

 numpy.zeros(20,20,20,30,30,30, dtype='float32')

which will halve your memory by using single-point (32-bit) floats. By default, numpy uses dtype = 'float64'.

Roughly speaking, a 32-bit float has 8 digits of accuracy, a 64-bit float has 16 digits of accuracy. This means that 1 + 1e-8 is displayed as 1 for 32-bit floats, and 1 + 1e-16 is displayed as 1 for 64-bit floats, but 1 + 1e-15 looks like 1.000000000000001 with 64-bit floats, but not 32-bit floats.

+6
source

More articles:


All Articles