np.nan has type float: arrays containing it must also have this datatype (or the complex or object datatype) so you may need to cast arr before you try to assign this value.
The error arises because the string value 'nan' can't be converted to an integer type to match arr's type.
>>> arr = arr.astype('float')
>>> arr[arr == 0] = 'nan' # or use np.nan
>>> arr
array([[ nan, 1., 2.],
[ 3., 4., 5.]])
Answer from Alex Riley on Stack Overflownp.nan has type float: arrays containing it must also have this datatype (or the complex or object datatype) so you may need to cast arr before you try to assign this value.
The error arises because the string value 'nan' can't be converted to an integer type to match arr's type.
>>> arr = arr.astype('float')
>>> arr[arr == 0] = 'nan' # or use np.nan
>>> arr
array([[ nan, 1., 2.],
[ 3., 4., 5.]])
An integer array can't hold a NaN value, so a new copy will have to be created anyway; so numpy.where may be used here to replace the values that satisfy the condition by NaN:
arr = np.arange(6).reshape(2, 3)
arr = np.where(arr==0, np.nan, arr)
# array([[nan, 1., 2.],
# [ 3., 4., 5.]])
A[A==NDV]=numpy.nan
A==NDV will produce a boolean array that can be used as an index for A
You can also use np.where to replace a number with NaN.
arr = np.where(arr==NDV, np.nan, arr)
For example, the following result can be obtained via
arr = np.array([[1, 1, 2], [2, 0, 1]])
arr = np.where(arr==1, np.nan, arr)

This creates a new copy (unlike A[A==NDV]=np.nan) but in some cases that could be useful. For example, if the array was initially an int dtype, it will have to converted into a float array anyway (because replacing values with NaN won't work otherwise) and np.where can handle that.
Randomly replace values in a numpy array
# The dataset
data = pd.read_csv('iris.data')
mat = data.iloc[:,:4].as_matrix()
Set the number of values to replace. For example 20%:
# Edit: changed len(mat) for mat.size
prop = int(mat.size * 0.2)
Randomly choose indices of the numpy array:
i = [random.choice(range(mat.shape[0])) for _ in range(prop)]
j = [random.choice(range(mat.shape[1])) for _ in range(prop)]
Change values with NaN
mat[i,j] = np.NaN
Dropout for any array dimension
Another way to do that with an array of more than 2 dimensions would be to use the numpy.put() function:
import numpy as np
import random
from sklearn import datasets
data = datasets.load_iris()['data']
def dropout(a, percent):
# create a copy
mat = a.copy()
# number of values to replace
prop = int(mat.size * percent)
# indices to mask
mask = random.sample(range(mat.size), prop)
# replace with NaN
np.put(mat, mask, [np.NaN]*len(mask))
return mat
This function returns a modified array:
modified = dropout(data, 0.2)
We can verify that the correct number of values have been modified:
np.sum(np.isnan(modified))/float(data.size)
[out]:
0.2
Depending on the data structure you are keeping the values there might be different solutions.
If you are using Numpy arrays, you can employ np.insert method which is referred here:
import numpy as np
a = np.arrray([(122.0, 1.0, -47.0), (123.0, 1.0, -47.0), (125.0, 1.0, -44.0)]))
np.insert(a, 2, np.nan, axis=0)
array([[ 122., 1., -47.],
[ 123., 1., -47.],
[ nan, nan, nan],
[ 125., 1., -44.]])
If you are using Pandas you can use instance method replace on the objects of the DataFrames as referred here:
In [106]:
df.replace('N/A',np.NaN)
Out[106]:
x y
0 10 12
1 50 11
2 18 NaN
3 32 13
4 47 15
5 20 NaN
In the code above, the first argument can be your arbitrary input which you want to change.
np.array([2,1,0,7]).dtype
# => dtype('int64')
np.array([2,1,np.nan,7]).dtype
# => dtype('float64')
Integers don't have a NaN value; np.nan is a float, and can't be placed in an integral array.
roganjosh beat me to the solution though :P (i.e. convert to a float array before trying to put a np.nan in there). With roganjosh's permission:
a = np.array([2,1,0,7])
a = a.astype(np.float64)
a[a == 0] = np.nan
a
# => array([ 2., 1., nan, 7.])
The reason why converting to list works is the fact that lists, unlike numpy arrays, don't mind mixing element types. When you re-convert a mixed list of integers and floats, numpy will assume you wanted a float array.
As mentioned in the comments, the problem is that a is of type int, and np.nan is of type float, so it cannot by directly assigned nor cast to int since it is only defined for float types.
You'd have to initialize an array of floats for the slice assignment to work:
a = np.array([2,1,0,7], dtype=float)
a[a==0] = np.nan
print(a)
# array([ 2., 1., nan, 7.])
You can't use numpy vector boolean indexing or comparisons with a Python list, so your l == 0 is simply giving you 0. It's only a fluke it works even partially, because you happened to have 0 as the first element.
>>> l = [0,2,3,4,0]
>>> l == 0
False
>>> int(l == 0)
0
>>> l[l == 0] = 999
>>> l
[999, 2, 3, 4, 0]
You could use a list comprehension and a ternary:
>>> l = [0,2,3,4,0]
>>> l = [np.nan if x == 0 else x for x in l]
>>> l
[nan, 2, 3, 4, nan]
(If you were working with numpy arrays, vector comparisons and assignments would work, but you can't mix integers and floats (such as np.nan) too nicely.)
You can just use the built in map:
l = map(lambda number: np.nan if number == 0 else number, l)
Which is equivalent to:
for i in range(len(l)):
l[i] = 0 if l[i] == np.nan else l[i]
Where A is your 2D array:
import numpy as np
A[np.isnan(A)] = 0
The function isnan produces a bool array indicating where the NaN values are. A boolean array can by used to index an array of the same shape. Think of it like a mask.
This should work:
from numpy import *
a = array([[1, 2, 3], [0, 3, NaN]])
where_are_NaNs = isnan(a)
a[where_are_NaNs] = 0
In the above case where_are_NaNs is:
In [12]: where_are_NaNs
Out[12]:
array([[False, False, False],
[False, False, True]], dtype=bool)
A complement about efficiency. The examples below were run with numpy 1.21.2
>>> aa = np.random.random(1_000_000)
>>> a = np.where(aa < 0.15, np.nan, aa)
>>> %timeit a[np.isnan(a)] = 0
536 µs ± 8.11 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
>>> a = np.where(aa < 0.15, np.nan, aa)
>>> %timeit np.where(np.isnan(a), 0, a)
2.38 ms ± 27.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
>>> a = np.where(aa < 0.15, np.nan, aa)
>>> %timeit np.nan_to_num(a, copy=True)
8.11 ms ± 401 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
>>> a = np.where(aa < 0.15, np.nan, aa)
>>> %timeit np.nan_to_num(a, copy=False)
3.8 ms ± 70.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
In consequence a[np.isnan(a)] = 0 is faster.