Updated answer, April 2025:
pd.to_numeric can convert arguments to a numeric type. The option errors='coerce' sets things to NaN. However, it can only work on 1D objects (i.e. scalar, list, tuple, 1-d array, or Series). Therefore, to use it on a DataFrame, we need to use df.apply to convert each column individually. Note that any **kwargs given to apply will be passed onto the function, so we can still set errors='coerce'.
Using pd.to_numeric along with df.apply will set any strings to NaN. If we want to convert those to 0 values, we can then use .fillna(0) on the resulting DataFrame.
For example (and note this also works with the strings suggested by the original question "$-" and "($24)"):
import pandas as pd
df = pd.DataFrame({
'a': (1, 'sd', 1),
'b': (2., 2., 'fg'),
'c': (4, "$-", "($24)")
})
print(df)
# a b c
# 0 1 2.0 4
# 1 sd 2.0 $-
# 2 1 fg ($24)
df = df.apply(pd.to_numeric, errors='coerce').fillna(0)
print(df)
# a b c
# 0 1.0 2.0 4.0
# 1 0.0 2.0 0.0
# 2 1.0 0.0 0.0
My original answer from 2015, which is now deprecated
You can use the convert_objects method of the DataFrame, with convert_numeric=True to change the strings to NaNs
From the docs:
convert_numeric: If True, attempt to coerce to numbers (including strings), with unconvertible values becoming NaN.
In [17]: df
Out[17]:
a b c
0 1. 2. 4
1 sd 2. 4
2 1. fg 5
In [18]: df2 = df.convert_objects(convert_numeric=True)
In [19]: df2
Out[19]:
a b c
0 1 2 4
1 NaN 2 4
2 1 NaN 5
Finally, if you want to convert those NaNs to 0's, you can use df.replace
In [20]: df2.replace('NaN',0)
Out[20]:
a b c
0 1 2 4
1 0 2 4
2 1 0 5
Answer from tmdavison on Stack OverflowUpdated answer, April 2025:
pd.to_numeric can convert arguments to a numeric type. The option errors='coerce' sets things to NaN. However, it can only work on 1D objects (i.e. scalar, list, tuple, 1-d array, or Series). Therefore, to use it on a DataFrame, we need to use df.apply to convert each column individually. Note that any **kwargs given to apply will be passed onto the function, so we can still set errors='coerce'.
Using pd.to_numeric along with df.apply will set any strings to NaN. If we want to convert those to 0 values, we can then use .fillna(0) on the resulting DataFrame.
For example (and note this also works with the strings suggested by the original question "$-" and "($24)"):
import pandas as pd
df = pd.DataFrame({
'a': (1, 'sd', 1),
'b': (2., 2., 'fg'),
'c': (4, "$-", "($24)")
})
print(df)
# a b c
# 0 1 2.0 4
# 1 sd 2.0 $-
# 2 1 fg ($24)
df = df.apply(pd.to_numeric, errors='coerce').fillna(0)
print(df)
# a b c
# 0 1.0 2.0 4.0
# 1 0.0 2.0 0.0
# 2 1.0 0.0 0.0
My original answer from 2015, which is now deprecated
You can use the convert_objects method of the DataFrame, with convert_numeric=True to change the strings to NaNs
From the docs:
convert_numeric: If True, attempt to coerce to numbers (including strings), with unconvertible values becoming NaN.
In [17]: df
Out[17]:
a b c
0 1. 2. 4
1 sd 2. 4
2 1. fg 5
In [18]: df2 = df.convert_objects(convert_numeric=True)
In [19]: df2
Out[19]:
a b c
0 1 2 4
1 NaN 2 4
2 1 NaN 5
Finally, if you want to convert those NaNs to 0's, you can use df.replace
In [20]: df2.replace('NaN',0)
Out[20]:
a b c
0 1 2 4
1 0 2 4
2 1 0 5
Use .to_numeric to covert the strings to numeric (set strings to NaN using the errors option 'coerce'):
df = pd.to_numeric(df, errors='coerce')
and then convert the NaN value to zeros using replace:
df.replace('NaN',0)
python - Pandas: Replacing Non-numeric cells with 0 - Stack Overflow
[Pandas] Replacing Zero Values in a Column
python - Pandas: How to replace Zero values in a column with the mean of that column, For all columns with Zero Value - Stack Overflow
python - Pandas: replace empty cell to 0 - Stack Overflow
You can use the to_numeric method, but it's not changing the value in place. You need to set the column to the new values:
training_data['usagequantity'] = (
pd.to_numeric(training_data['usagequantity'],
errors='coerce')
.fillna(0)
)
to_numeric sets the non-numeric values to NaNs, and then the chained fillna method replaces the NaNs with zeros.
Following code can work:
df.col =pd.to_numeric(df.col, errors ='coerce').fillna(0).astype('int')
Hi all,
I decided to take my first try at a kaggle competition, however, I've been struggling something for awhile now. Perhaps you can help.
Basically, I've got a dataframe where the latitude and longitude (floats) are both zero for a very very small number of lines.
The std deviation for these columns is tiny, so I was just going to replace the zero values with the mean values. How should I go about this? Nothing I have tried so far has worked.
Thanks.
So this is pandas dataframe I will using mask make all 0 to np.nan , then fillna
df=df.mask(df==0).fillna(df.mean())
Same we can achieve directly using replace method. Without fillna
df.replace(0,df.mean(axis=0),inplace=True)
Method info: Replace values given in "to_replace" with "value".
Values of the DataFrame are replaced with other values dynamically. This differs from updating with .loc or .iloc which require you to specify a location to update with some value.
You are creating a copy of the dataframe but the original one is not keeping the changes, you need to specify "inplace=True" if you want the dataframe to persist the changes
result.fillna(0, inplace=True)
If the empty field means '' (empty sign) then you can use:
dataframe['column_name'].replace('',0)
That creates a new series with the replaced values, so to update the original dataframe do:
dataframe['column_name'] = dataframe['column_name'].replace('',0)
