To remove NaN values from a NumPy array x:
x = x[~numpy.isnan(x)]
Explanation
The inner function numpy.isnan returns a boolean/logical array which has the value True everywhere that x is not-a-number. Since we want the opposite, we use the logical-not operator ~ to get an array with Trues everywhere that x is a valid number.
Lastly, we use this logical array to index into the original array x, in order to retrieve just the non-NaN values.
python - How to remove all rows in a numpy.ndarray that contain non-numeric values - Stack Overflow
python - How can I remove rows from a numpy array that have NaN as the first element? - Stack Overflow
python - Numpy: Drop rows with all nan or 0 values - Stack Overflow
How should I initialize a numpy array of NaN values?
To remove NaN values from a NumPy array x:
x = x[~numpy.isnan(x)]
Explanation
The inner function numpy.isnan returns a boolean/logical array which has the value True everywhere that x is not-a-number. Since we want the opposite, we use the logical-not operator ~ to get an array with Trues everywhere that x is a valid number.
Lastly, we use this logical array to index into the original array x, in order to retrieve just the non-NaN values.
filter(lambda v: v==v, x)
works both for lists and numpy array since v!=v only for NaN
>>> a = np.array([[1,2,3], [4,5,np.nan], [7,8,9]])
array([[ 1., 2., 3.],
[ 4., 5., nan],
[ 7., 8., 9.]])
>>> a[~np.isnan(a).any(axis=1)]
array([[ 1., 2., 3.],
[ 7., 8., 9.]])
and reassign this to a.
Explanation: np.isnan(a) returns a similar array with True where NaN, False elsewhere. .any(axis=1) reduces an m*n array to n with an logical or operation on the whole rows, ~ inverts True/False and a[ ] chooses just the rows from the original array, which have True within the brackets.
You can also use a masked array via np.ma.fix_invalid to create a mask and filter out "bad" values (such as NaN, inf).
arr = np.array([
[0, 1, np.inf],
[2.2, 3.3, 4.],
[np.nan, 5.5, 6],
[7.8, -np.inf, 9.9],
[10, 11, 12]
])
new_arr = arr[~np.ma.fix_invalid(arr).mask.any(axis=1)]
# array([[ 2.2, 3.3, 4. ],
# [10. , 11. , 12. ]])
If the array contains strings such as 'NA', then np.where may be useful to "mask" these values and then filter them out.
arr = np.array([
[0, 1, 'N/A'],
[2.2, 3.3, 4.],
[np.nan, 5.5, 6],
[7.8, 'NA', 9.9],
[10, 11, 12]
], dtype=object)
tmp = np.where(np.isin(arr, ['NA', 'N/A']), np.nan, arr).astype(float)
new_arr = tmp[~np.isnan(tmp).any(axis=1)]
# array([[ 2.2, 3.3, 4. ],
# [10. , 11. , 12. ]])
import numpy as np
a = np.array([
[1, 0, 0],
[0, np.nan, 0],
[0, 0, 0],
[np.nan, np.nan, np.nan],
[2, 3, 4]
])
mask = np.all(np.isnan(a) | np.equal(a, 0), axis=1)
a[~mask]
This will remove all rows which are all zeros, or all nans:
mask = np.all(np.isnan(arr), axis=1) | np.all(arr == 0, axis=1)
arr = arr[~mask]
And this will remove all rows which are all either zeros or nans:
mask = np.all(np.isnan(arr) | arr == 0, axis=1)
arr = arr[~mask]