Geek Answers Handbook

Average values ​​depending on binning relative to the second variable

I am working with python / numpy. As input, I have a large number of pairs of values (x,y) . I basically want to build <y>(x) , i.e. The average y for a specific data buffer x . Right now I'm using a simple for loop to achieve this, which is very slow.

 # create example data x = numpy.random.rand(1000) y = numpy.random.rand(1000) # set resolution xbins = 100 # find x bins H, xedges, yedges = numpy.histogram2d(x, y, bins=(xbins,xbins) ) # calculate mean and std of y for each x bin mean = numpy.zeros(xbins) std = numpy.zeros(xbins) for i in numpy.arange(xbins): mean[i] = numpy.mean(y[ numpy.logical_and( x>=xedges[i], x<xedges[i+1] ) ]) std[i] = numpy.std (y[ numpy.logical_and( x>=xedges[i], x<xedges[i+1] ) ])

Is it possible to have some kind of vectorized record for him?

+4
source share
2 answers

You unnecessarily complicate things. All you need to know is for each bin in x that n , sy and sy2 , the number of y values ​​in this buffer x , the sum of these y values ​​and the sum of their squares. You can get them as:

 >>> n, _ = np.histogram(x, bins=xbins) >>> sy, _ = np.histogram(x, bins=xbins, weights=y) >>> sy2, _ = np.histogram(x, bins=xbins, weights=y*y)

From:

 >>> mean = sy / n >>> std = np.sqrt(sy2/n - mean*mean)
+13
source

If you can use pandas:

 import pandas as pd xedges = np.linspace(x.min(), x.max(), xbins+1) xedges[0] -= 0.00001 xedges[-1] += 0.000001 c = pd.cut(x, xedges) g = pd.groupby(pd.Series(y), c.labels) mean2 = g.mean() std2 = g.std(0)
0
source

More articles:


All Articles