Implementing numpy in1d for 2D arrays?

Question

I have a two-dimensional numpy S array representing a state space, with 80,000,000 rows (as states) and 5 columns (as state variables).

I initialize K0 with S, and at each iteration I apply the state transition function f (x) to all states in Ki and delete the states, f (x) does not belong to Ki, which leads to Ki + 1. Until it converges, those. Ki + 1 = Ki.

This will require age:

K = S to_delete = [0] While to_delete: to_delete = [] for i in xrange(len(K)): if not f(i) in K: to_delete.append(K(i)) K = delete(K,to_delete,0) 

So, I wanted to make a vectorized implementation:

slice K in the columns, apply f and, join them again, thus obtaining f (K).

Now the question is how to get an array of length len (K), for example Sel, where each line of Sel [i] determines whether f (K [i]) is in K. Just like the function in1d works.

Then it would be easy to do

 K=K[Sel]] 

Answer 1

Your question is hard to understand because it contains extraneous information and contains typos. If I understand correctly, you just need an effective way to perform a dial operation on the lines of a 2D array (in this case, the intersection of the lines K and f(K) ).

You can do this with numpy.in1d if you create a structured array.

code:

if it is K :

 In [50]: k Out[50]: array([[6, 6], [3, 7], [7, 5], [7, 3], [1, 3], [1, 5], [7, 6], [3, 8], [6, 1], [6, 0]]) 

and this is f(K) (for this example, I subtract 1 from the first column and add 1 to the second):

 In [51]: k2 Out[51]: array([[5, 7], [2, 8], [6, 6], [6, 4], [0, 4], [0, 6], [6, 7], [2, 9], [5, 2], [5, 1]]) 

you can find all lines in K also found in f(K) by doing something like the following:

 In [55]: k[np.in1d(k.view(dtype='i,i').reshape(k.shape[0]),k2.view(dtype='i,i'). reshape(k2.shape[0]))] Out[55]: array([[6, 6]]) 

view and reshape create flat structured views so that each row is displayed as one element in in1d . in1d creates a logical index K matching elements, which is used to display the index K and returns a filtered array.

Answer 2

The above answer is great.

But if you do not want to mix with structured arrays and want a solution that does not care about what type of your array, as well as the sizes of the elements of the array , I came up with this:

 k[np.in1d(list(map(np.ndarray.dumps, k)), list(map(np.ndarray.dumps, k2)))] 

basically list(map(np.ndarray.dumps, k)) instead of k.view(dtype='f8,f8').reshape(k.shape[0]) .

Note that this solution is 50 times slower.

 k = np.array([[6.5, 6.5], [3.5, 7.5], [7.5, 5.5], [7.5, 3.5], [1.5, 3.5], [1.5, 5.5], [7.5, 6.5], [3.5, 8.5], [6.5, 1.5], [6.5, 0.5]]) k = np.tile(k, (1000, 1)) k2 = np.c_[k[:, 0] - 1, k[:, 1] + 1] In [132]: k.shape, k2.shape Out[132]: ((10000, 2), (10000, 2)) In [133]: timeit k[np.in1d(k.view(dtype='f8,f8').reshape(k.shape[0]),k2.view(dtype='f8,f8').reshape(k2.shape[0]))] 10 loops, best of 3: 22.2 ms per loop In [134]: timeit k[np.in1d(list(map(np.ndarray.dumps, k)), list(map(np.ndarray.dumps, k2)))] 1 loop, best of 3: 892 ms per loop 

It may be extreme for small entries, but for op, it takes 1h 20min instead of 2 minutes.

Answer 3

I'm not sure if I fully understand your question, but if Pavel’s interpretation is correct, it can be effectively and fully resolved using numpy_indexed as such in one readable line:

 import numpy_indexed as npi K = npi.intersection(K, f(K)) 

In addition, this works for strings of any type or form.