Efficient array threshold filter with numpy

Question

I need to filter an array to remove elements that are below a certain threshold. My current code is as follows:

threshold = 5 a = numpy.array(range(10)) # testing data b = numpy.array(filter(lambda x: x >= threshold, a)) 

The problem is that this creates a temporary list using a filter with a lambda function (slow).

Since this is a fairly simple operation, there may be a numpy function that does this in an efficient way, but I could not find it.

I thought that another way to achieve this could be to sort the array, search for the threshold index and move the slice from this index forward, but even if it is faster for small inputs (and it will not be noticeable in any case), it is finally asymptotically less efficient as input size increases.

Any ideas? Thank!

Update . I also made some measurements, and sorting + slicing was twice as fast as a clean python filter when the input was 100,000,000 records.

 In [321]: r = numpy.random.uniform(0, 1, 100000000) In [322]: %timeit test1(r) # filter 1 loops, best of 3: 21.3 s per loop In [323]: %timeit test2(r) # sort and slice 1 loops, best of 3: 11.1 s per loop In [324]: %timeit test3(r) # boolean indexing 1 loops, best of 3: 1.26 s per loop 

Answer 1

b = a[a>threshold] this should do

I tested as follows:

 import numpy as np, datetime # array of zeros and ones interleaved lrg = np.arange(2).reshape((2,-1)).repeat(1000000,-1).flatten() t0 = datetime.datetime.now() flt = lrg[lrg==0] print datetime.datetime.now() - t0 t0 = datetime.datetime.now() flt = np.array(filter(lambda x:x==0, lrg)) print datetime.datetime.now() - t0 

I got

 $ python test.py 0:00:00.028000 0:00:02.461000 

http://docs.scipy.org/doc/numpy/user/basics.indexing.html#boolean-or-mask-index-arrays

Answer 2

To add to

@yosukesabai

answer, it is important to use a diffrent variable, as this will return an empty array:

 im=im[im>167] 

I can’t explain why, although, perhaps, because they, too, are tired of thinking :(