In case you want to normalize all of the nulls with python's None.
df.fillna(np.nan).replace([np.nan], [None])
The first fillna will replace all of (None, NAT, np.nan, etc) with Numpy's NaN, then replace Numpy's NaN with python's None.
In case you want to normalize all of the nulls with python's None.
df.fillna(np.nan).replace([np.nan], [None])
The first fillna will replace all of (None, NAT, np.nan, etc) with Numpy's NaN, then replace Numpy's NaN with python's None.
Setup
Consider the sample dataframe df
df = pd.DataFrame(dict(A=[1, None], B=[None, 2], C=[None, 'D']))
df
A B C
0 1.0 NaN None
1 NaN 2.0 D
I can confirm the error
df.fillna(dict(A=1, B=None, C=4))
ValueError: must specify a fill method or value
This happens because pandas is cycling through keys in the dictionary and executing a fillna for each relevant column. If you look at the signature of the pd.Series.fillna method
Series.fillna(value=None, method=None, axis=None, inplace=False, limit=None, downcast=None, **kwargs)
You'll see the default value is None. So we can replicate this error with
df.A.fillna(None)
Or equivalently
df.A.fillna()
I'll add that I'm not terribly surprised considering that you are attempting to fill a null value with a null value.
What you need is a work around
Solution
Use pd.DataFrame.fillna over columns that you want to fill with non-null values. Then follow that up with a pd.DataFrame.replace on the specific columns you want to swap one null value with another.
df.fillna(dict(A=1, C=2)).replace(dict(B={np.nan: None}))
A B C
0 1.0 None 2
1 1.0 2 D
BUG: fillna() doesn't work with `value=None` without method specified, but method is deprecated
Can't fillna with None in a dataframe
Issue replacing 'None' with df.fillna()
How to fillna with None in a pandas dataframe?
Videos
Pretty straightforward, I have a dataframe that has columns with different mixtures of np.nan and None as the "null" value for that column. I want to get them all to be "None", but
df.fillna(None) df.replace(np.nan, None) df.fillna(values=None)
all don't work. Anyone run into this issue before?
Also why the bloody fucking hell does
np.nan==np.nan
evaluate to false? (Yes I know its by definition, but some bloke thought it was a good idea to write the definition that way.) Is that actually useful for anything? In this particular case, it makes me want to use None instead.
Hey All,
I am using the below code to export Jira issues. Not all of my issues have all the custom fields I am trying to export so they end up being 'None'. I would like to replace the 'None' with an empty string. I am trying to use pandas df.fillna but it's not doing what I thought it would. In spite of using fillna, my output is unchanged and still contains 'None' values.
What am I doing wrong?
Thanks!
Code:
# Search all issues, and time execution
jira_issues = jira.search_issues(jql,maxResults=50)
# Converge JSON to Pandas DataFrame
for issue in jira_issues:
try:
issue_fields = pd.DataFrame({
'id': [issue.id],
'Due date': str(issue.fields.duedate),
'Actual Start Date': str(issue.fields.customfield_10055),
'Actual Completion Date': str(issue.fields.customfield_10057)
})
except AttributeError:
pass
issues = pd.concat([issues,issue_fields])
issues.fillna("",inplace=True)
print(issues)Output:
0 79906 None None None 0 79904 2022-07-07 2022-05-18 2022-05-25 0 79903 2022-04-13 2022-04-22 2022-04-22 0 79902 2022-06-15 2022-06-04 2022-06-04 0 79901 None None None 0 79900 None 2022-06-14 None 0 79899 2022-05-06 2022-05-02 2022-05-06 0 79897 None None None 0 79896 None None None
Checking column types using dtypes:
id object Due date object Actual Start Date object Actual Completion Date object dtype: object