Python binning data with scipy / numpy

Question

Is there a more efficient way to get the average value of an array in predefined cells? for example, do I have an array of numbers and an array corresponding to the start and end positions of bin in this array, and I want to just take the average value in these cells? I have a code that does this below, but I wonder how it can be shortened and improved. thank.

from scipy import * from numpy import * def get_bin_mean(a, b_start, b_end): ind_upper = nonzero(a >= b_start)[0] a_upper = a[ind_upper] a_range = a_upper[nonzero(a_upper < b_end)[0]] mean_val = mean(a_range) return mean_val data = rand(100) bins = linspace(0, 1, 10) binned_data = [] n = 0 for n in range(0, len(bins)-1): b_start = bins[n] b_end = bins[n+1] binned_data.append(get_bin_mean(data, b_start, b_end)) print binned_data 

Answer 1

Probably faster and easier to use numpy.digitize() :

 import numpy data = numpy.random.random(100) bins = numpy.linspace(0, 1, 10) digitized = numpy.digitize(data, bins) bin_means = [data[digitized == i].mean() for i in range(1, len(bins))] 

An alternative to this is to use numpy.histogram() :

 bin_means = (numpy.histogram(data, bins, weights=data)[0] / numpy.histogram(data, bins)[0]) 

Try it yourself, which one is faster ... :)

Answer 2

The Scipy function (> = 0.11) scipy.stats.binned_statistic specifically addresses the above issue.

In the same example as in previous answers, a Scipy solution would be

 import numpy as np from scipy.stats import binned_statistic data = np.random.rand(100) bin_means = binned_statistic(data, data, bins=10, range=(0, 1))[0] 

Answer 3

I do not know why this thread became necrotic; but here is the approved answer of 2014, which should be much faster:

 import numpy as np data = np.random.rand(100) bins = 10 slices = np.linspace(0, 100, bins+1, True).astype(np.int) counts = np.diff(slices) mean = np.add.reduceat(data, slices[:-1]) / counts print mean 

Answer 4

The numpy_indexed package (disclaimer: I am the author of it) contains functionality for efficiently performing operations of this type:

 import numpy_indexed as npi print(npi.group_by(np.digitize(data, bins)).mean(data)) 

This is essentially the same solution as the previous one; but now wrapped in a nice interface, with tests and that's it :)

Answer 5

I would add, as well, to answer the question of finding bin averages using python histogram2d , that scipy also has a function specifically designed to calculate two-dimensional bin statistics for one or more data sets

 import numpy as np from scipy.stats import binned_statistic_2d x = np.random.rand(100) y = np.random.rand(100) values = np.random.rand(100) bin_means = binned_statistic_2d(x, y, values, bins=10).statistic 

the scipy.stats.binned_statistic_dd function is a generalization of this function for higher data sets