To remove NaN values from a NumPy array x:
x = x[~numpy.isnan(x)]
Explanation
The inner function numpy.isnan returns a boolean/logical array which has the value True everywhere that x is not-a-number. Since we want the opposite, we use the logical-not operator ~ to get an array with Trues everywhere that x is a valid number.
Lastly, we use this logical array to index into the original array x, in order to retrieve just the non-NaN values.
To remove NaN values from a NumPy array x:
x = x[~numpy.isnan(x)]
Explanation
The inner function numpy.isnan returns a boolean/logical array which has the value True everywhere that x is not-a-number. Since we want the opposite, we use the logical-not operator ~ to get an array with Trues everywhere that x is a valid number.
Lastly, we use this logical array to index into the original array x, in order to retrieve just the non-NaN values.
filter(lambda v: v==v, x)
works both for lists and numpy array since v!=v only for NaN
>>> a = np.array([[1,2,3], [4,5,np.nan], [7,8,9]])
array([[ 1., 2., 3.],
[ 4., 5., nan],
[ 7., 8., 9.]])
>>> a[~np.isnan(a).any(axis=1)]
array([[ 1., 2., 3.],
[ 7., 8., 9.]])
and reassign this to a.
Explanation: np.isnan(a) returns a similar array with True where NaN, False elsewhere. .any(axis=1) reduces an m*n array to n with an logical or operation on the whole rows, ~ inverts True/False and a[ ] chooses just the rows from the original array, which have True within the brackets.
You can also use a masked array via np.ma.fix_invalid to create a mask and filter out "bad" values (such as NaN, inf).
arr = np.array([
[0, 1, np.inf],
[2.2, 3.3, 4.],
[np.nan, 5.5, 6],
[7.8, -np.inf, 9.9],
[10, 11, 12]
])
new_arr = arr[~np.ma.fix_invalid(arr).mask.any(axis=1)]
# array([[ 2.2, 3.3, 4. ],
# [10. , 11. , 12. ]])
If the array contains strings such as 'NA', then np.where may be useful to "mask" these values and then filter them out.
arr = np.array([
[0, 1, 'N/A'],
[2.2, 3.3, 4.],
[np.nan, 5.5, 6],
[7.8, 'NA', 9.9],
[10, 11, 12]
], dtype=object)
tmp = np.where(np.isin(arr, ['NA', 'N/A']), np.nan, arr).astype(float)
new_arr = tmp[~np.isnan(tmp).any(axis=1)]
# array([[ 2.2, 3.3, 4. ],
# [10. , 11. , 12. ]])
import numpy as np
a = np.array([
[1, 0, 0],
[0, np.nan, 0],
[0, 0, 0],
[np.nan, np.nan, np.nan],
[2, 3, 4]
])
mask = np.all(np.isnan(a) | np.equal(a, 0), axis=1)
a[~mask]
This will remove all rows which are all zeros, or all nans:
mask = np.all(np.isnan(arr), axis=1) | np.all(arr == 0, axis=1)
arr = arr[~mask]
And this will remove all rows which are all either zeros or nans:
mask = np.all(np.isnan(arr) | arr == 0, axis=1)
arr = arr[~mask]
Dataframe:
values_1 values_2
0 700.0 NaN
1 NaN 150.0
2 500.0 350.0
3 NaN 400.0
4 1200.0 5000.0
You can try this:
df = df.dropna()
df = df.reset_index(drop=True)
Outputs:
values_1 values_2
0 500.0 350.0
1 1200.0 5000.0
Given numpy array as
import numpy as np
input = np.array([[1,1,0,np.nan], [1,1,1,30], [1,0,1,np.nan]])
try
output = input[~np.isnan(input).any(axis=1)]
gives output as
[[ 1. 1. 1. 30.]]