np.count_nonzero(~np.isnan(data))
~ inverts the boolean matrix returned from np.isnan.
np.count_nonzero counts values that is not 0\false. .sum should give the same result. But maybe more clearly to use count_nonzero
Testing speed:
In [23]: data = np.random.random((10000,10000))
In [24]: data[[np.random.random_integers(0,10000, 100)],:][:, [np.random.random_integers(0,99, 100)]] = np.nan
In [25]: %timeit data.size - np.count_nonzero(np.isnan(data))
1 loops, best of 3: 309 ms per loop
In [26]: %timeit np.count_nonzero(~np.isnan(data))
1 loops, best of 3: 345 ms per loop
In [27]: %timeit data.size - np.isnan(data).sum()
1 loops, best of 3: 339 ms per loop
data.size - np.count_nonzero(np.isnan(data)) seems to barely be the fastest here. other data might give different relative speed results.
np.count_nonzero(~np.isnan(data))
~ inverts the boolean matrix returned from np.isnan.
np.count_nonzero counts values that is not 0\false. .sum should give the same result. But maybe more clearly to use count_nonzero
Testing speed:
In [23]: data = np.random.random((10000,10000))
In [24]: data[[np.random.random_integers(0,10000, 100)],:][:, [np.random.random_integers(0,99, 100)]] = np.nan
In [25]: %timeit data.size - np.count_nonzero(np.isnan(data))
1 loops, best of 3: 309 ms per loop
In [26]: %timeit np.count_nonzero(~np.isnan(data))
1 loops, best of 3: 345 ms per loop
In [27]: %timeit data.size - np.isnan(data).sum()
1 loops, best of 3: 339 ms per loop
data.size - np.count_nonzero(np.isnan(data)) seems to barely be the fastest here. other data might give different relative speed results.
Quick-to-write alternative
Even though is not the fastest choice, if performance is not an issue you can use:
sum(~np.isnan(data)).
Performance:
In [7]: %timeit data.size - np.count_nonzero(np.isnan(data))
10 loops, best of 3: 67.5 ms per loop
In [8]: %timeit sum(~np.isnan(data))
10 loops, best of 3: 154 ms per loop
In [9]: %timeit np.sum(~np.isnan(data))
10 loops, best of 3: 140 ms per loop
You can add axis=0 to count null per column:
number_nan_in_arr = np.count_nonzero(np.isnan(arr), axis=0)
Example:
import numpy as np
a = np.array([[0, np.nan, 7, 0],
[3, 0, 2, np.nan]])
np.count_nonzero(np.isnan(a), axis=0)
Output:
array([0, 1, 0, 1], dtype=int64)
Simply sum the True values:
number_nan_in_arr = np.isnan(arr).sum(0)