Dictionary 4-D numpy array: how to optimize?

Question

I had a problem performing initialization of a four-dimensional tensor tensor dictionary.

I have a list of coefficient names:

cnames = ['CN', 'CM', 'CA', 'CY', 'CLN' ...]; 

which is not fixed (depends on the top code). For each coefficient, I have to generate the four-dimensional tensor [nalpha X nmach X nbeta X nalt] of zeros (for the purpose of preassociation), so I do:

 #Number of coefficients numofc = len(cnames); final_data = {}; #I have to generate <numofc> 4D matrixes for i in range(numofc): final_data[cnames[i]]=n.zeros((nalpha,nmach,nbeta,nalt)); 

each index is an integer from 10 to 30. each index is an integer from 100 to 200

It takes 4 minutes. How can I speed this up? Or am I doing something wrong?

Answer 1

The code you sent should not take 4 minutes (if cnames not very large or you have very little RAM and it is forced to use the swap space).

 import numpy as np cnames = ['CN', 'CM', 'CA', 'CY', 'CLN']*1000 nalpha,nmach,nbeta,nalt = 10,20,30,40 #Number of coefficients numofc = len(cnames) final_data = {} #I have to generate <numofc> 4D matrixes for i in range(numofc): final_data[cnames[i]]=np.zeros((nalpha,nmach,nbeta,nalt)) 

Even if cnames has 5000 elements, it should still only take a couple of seconds:

 % time test.py real 0m4.559s user 0m0.856s sys 0m3.328s 

The semicolons at the end of the statements indicate that you have experience in some other language. Be careful when translating commands from this language line by line into NumPy / Python. Coding in NumPy, like in C, is a recipe for slowness.

In particular, try to avoid updating elements in an element-element of an array. This works fine in C, but it works very slowly with Python. NumPy achieves speed by delegating functions encoded in Fortran or Cython or C or C ++. When updating arrays by elements, you use Python loops, which are not so fast.

Instead, try rephrasing your calculations in terms of operations on entire arrays (or at least slices of arrays).

I probably thought too much about the cause of the problem. You need a profile of your code , and then, if you want more specific help, post the profile result plus the problem code (most useful in the SSCCE form).