You can create a masked array like this:
data = np.array([[1,2,3], [4,5,np.NaN], [np.NaN,6,np.NaN], [0,0,0]])
masked_data = np.ma.masked_array(data, np.isnan(data))
# calculate your weighted average here instead
weights = [1, 1, 1]
average = np.ma.average(masked_data, axis=1, weights=weights)
# this gives you the result
result = average.filled(np.nan)
print(result)
This outputs:
[ 2. 4.5 6. 0. ]
Answer from Alex on Stack OverflowYou can create a masked array like this:
data = np.array([[1,2,3], [4,5,np.NaN], [np.NaN,6,np.NaN], [0,0,0]])
masked_data = np.ma.masked_array(data, np.isnan(data))
# calculate your weighted average here instead
weights = [1, 1, 1]
average = np.ma.average(masked_data, axis=1, weights=weights)
# this gives you the result
result = average.filled(np.nan)
print(result)
This outputs:
[ 2. 4.5 6. 0. ]
You can simply multiply the input array with the weights and sum along the specified axis ignoring NaNs with np.nansum. Thus, for your case, assuming the weights are to be used along axis = 1 on the input array sst_filt, the summations would be -
np.nansum(sst_filt*weights,axis=1)
Accounting for the NaNs while averaging, we will end up with :
def nanaverage(A,weights,axis):
return np.nansum(A*weights,axis=axis)/((~np.isnan(A))*weights).sum(axis=axis)
Sample run -
In [200]: sst_filt # 2D array case
Out[200]:
array([[ 0., 1.],
[ nan, 3.],
[ 4., 5.]])
In [201]: weights
Out[201]: array([ 0.25, 0.75])
In [202]: nanaverage(sst_filt,weights=weights,axis=1)
Out[202]: array([0.75, 3. , 4.75])
Alternatively, you can use a MaskedArray as such:
>>> import numpy as np >>> a = np.array([1,2,np.nan,4]) >>> weights = np.array([4,3,2,1]) >>> ma = np.ma.MaskedArray(a, mask=np.isnan(a)) >>> np.ma.average(ma, weights=weights) 1.75
First find out indices where the items are not nan, and then pass the filtered versions of a and weights to numpy.average:
>>> import numpy as np
>>> a = np.array([1, 2, np.nan,4])
>>> weights = np.array([4, 3, 2, 1])
>>> indices = np.where(np.logical_not(np.isnan(a)))[0]
>>> np.average(a[indices], weights=weights[indices])
1.75
As suggested by @mtrw in comments, it would be cleaner to use masked array here instead of index array:
>>> indices = ~np.isnan(a)
>>> np.average(a[indices], weights=weights[indices])
1.75
I think what you want is a masked array:
dat = np.array([[1,2,3], [4,5,'nan'], ['nan',6,'nan'], ['nan','nan','nan']])
mdat = np.ma.masked_array(dat,np.isnan(dat))
mm = np.mean(mdat,axis=1)
print mm.filled(np.nan) # the desired answer
Edit: Combining all of the timing data
from timeit import Timer
setupstr="""
import numpy as np
from scipy.stats.stats import nanmean
dat = np.random.normal(size=(1000,1000))
ii = np.ix_(np.random.randint(0,99,size=50),np.random.randint(0,99,size=50))
dat[ii] = np.nan
"""
method1="""
mdat = np.ma.masked_array(dat,np.isnan(dat))
mm = np.mean(mdat,axis=1)
mm.filled(np.nan)
"""
N = 2
t1 = Timer(method1, setupstr).timeit(N)
t2 = Timer("[np.mean([l for l in d if not np.isnan(l)]) for d in dat]", setupstr).timeit(N)
t3 = Timer("np.array([r[np.isfinite(r)].mean() for r in dat])", setupstr).timeit(N)
t4 = Timer("np.ma.masked_invalid(dat).mean(axis=1)", setupstr).timeit(N)
t5 = Timer("nanmean(dat,axis=1)", setupstr).timeit(N)
print 'Time: %f\tRatio: %f' % (t1,t1/t1 )
print 'Time: %f\tRatio: %f' % (t2,t2/t1 )
print 'Time: %f\tRatio: %f' % (t3,t3/t1 )
print 'Time: %f\tRatio: %f' % (t4,t4/t1 )
print 'Time: %f\tRatio: %f' % (t5,t5/t1 )
Returns:
Time: 0.045454 Ratio: 1.000000
Time: 8.179479 Ratio: 179.950595
Time: 0.060988 Ratio: 1.341755
Time: 0.070955 Ratio: 1.561029
Time: 0.065152 Ratio: 1.433364
If performance matters, you should use bottleneck.nanmean() instead:
http://pypi.python.org/pypi/Bottleneck
For me working implemented this solution:
def f(x):
indices = ~np.isnan(x)
return np.average(x[indices], weights=df.loc[x.index[indices], 'two'])
df = df.groupby('four').agg(sum=('two','sum'), weighted_avg=('one', f))
print (df)
sum weighted_avg
four
bar -2.942607 0.648173
foo 1.086501 -1.086525
EDIT:
def f(x):
indices = ~np.isnan(x)
if indices.all():
return np.average(x[indices], weights=df.loc[x.index[indices], 'two'])
else:
return np.nan
This appears to be more robust:
def f(x):
indices = (~np.isnan(x)) & (~np.isnan(df[weight_column]))[x.index]
try:
return np.average(x[indices], weights=df.loc[x.index[indices], weight_column])
except ZeroDivisionError:
return np.nan
df = df.groupby('four').agg(sum=('two','sum'), weighted_avg=('one', f))