pandas.isnull() (also pd.isna(), in newer versions) checks for missing values in both numeric and string/object arrays. From the documentation, it checks for:
NaN in numeric arrays, None/NaN in object arrays
Quick example:
import pandas as pd
import numpy as np
s = pd.Series(['apple', np.nan, 'banana'])
pd.isnull(s)
Out[9]:
0 False
1 True
2 False
dtype: bool
The idea of using numpy.nan to represent missing values is something that pandas introduced, which is why pandas has the tools to deal with it.
Datetimes too (if you use pd.NaT you won't need to specify the dtype)
In [24]: s = Series([Timestamp('20130101'),np.nan,Timestamp('20130102 9:30')],dtype='M8[ns]')
In [25]: s
Out[25]:
0 2013-01-01 00:00:00
1 NaT
2 2013-01-02 09:30:00
dtype: datetime64[ns]``
In [26]: pd.isnull(s)
Out[26]:
0 False
1 True
2 False
dtype: bool
Answer from Marius on Stack Overflowpandas.isnull() (also pd.isna(), in newer versions) checks for missing values in both numeric and string/object arrays. From the documentation, it checks for:
NaN in numeric arrays, None/NaN in object arrays
Quick example:
import pandas as pd
import numpy as np
s = pd.Series(['apple', np.nan, 'banana'])
pd.isnull(s)
Out[9]:
0 False
1 True
2 False
dtype: bool
The idea of using numpy.nan to represent missing values is something that pandas introduced, which is why pandas has the tools to deal with it.
Datetimes too (if you use pd.NaT you won't need to specify the dtype)
In [24]: s = Series([Timestamp('20130101'),np.nan,Timestamp('20130102 9:30')],dtype='M8[ns]')
In [25]: s
Out[25]:
0 2013-01-01 00:00:00
1 NaT
2 2013-01-02 09:30:00
dtype: datetime64[ns]``
In [26]: pd.isnull(s)
Out[26]:
0 False
1 True
2 False
dtype: bool
Is your type really arbitrary? If you know it is just going to be a int float or string you could just do
if val.dtype == float and np.isnan(val):
assuming it is wrapped in numpy , it will always have a dtype and only float and complex can be NaN
Your first problem is that math.isnan() does not accept string values as input. You can try and see math.isnan('any string').
Because you are in the pandas dataframe already, you are better off to use Pandas to handle your case. For example:
df.dropna() # column-wise nan drop
df.dropna(axis=1) # row-wise nan drop
Note that there are very useful arguments for additional convenience in dropna(), so do check those out from the doctring or corresponding manual entries.
As a word of advice for along the journey, when you work with pandas, it's good to keep in mind that often whatever you want to do, is easier to do just within native pandas functionality. Because Pandas is kind of a gold standard for this type of work, generally speaking whatever you are thinking you want to do (if it makes sense) the Pandas community had already thought of (and implemented) it.
Your given dataframe is tough to replicate. Here is a sample df:
df = pd.DataFrame([["11","1", np.nan], [np.nan,"1", "2"], ['abc','def','ijk']],
columns=["ix","a", "b"])
>>df
a b c
0 11 1 NaN
1 NaN 1 2
2 abc def ijk
from the docs : df.dropna()
df.dropna()
This returns all the rows which doesnot have any nan in any columns.
Output:
a b c
2 abc def ijk
for filtering columns which doesnot have any nan:
df.dropna(axis=1)
b
0 1
1 1
2 def
For finding rows with nan:
df_nan= df.drop(list(df.dropna().index))
Also please check the how= builtin function which allows you to drop na values for any or all rows/columns depending on the axis selected.
Just use both:
>>> names=['Pat','Sam', np.nan, 'Tom', '']
>>> for idx,name in enumerate(names):
... if name == '' or pd.isnull(name):
... print(idx)
...
2
4
However, realize that:
>>> pd.isnull(None)
True
So if you want to check specifically for NaN and not None, use math.isnan (while guarding against passing non-float values to math.isnan:
>>> import math
>>> for idx,name in enumerate(names):
... if name == '' or (isinstance(name, float) and math.isnan(name)):
... print(idx)
...
2
4
There is a way to combine options #1 and #2 and get the result you are looking for:
names = ['Pat', 'Sam', np.nan, 'Tom', '']
for idx, name in enumerate(names):
if not name or pd.isnull(name):
print(idx)
Use the pandas method isnull to test:
Use with AIIn [45]:
df = pd.DataFrame({'words':['from', 'fairest', 'creatures' ,'we' ,'desire', 'increase' ,nan ,'that' ,'thereby']})
df
Out[45]:
words
0 from
1 fairest
2 creatures
3 we
4 desire
5 increase
6 NaN
7 that
8 thereby
In [46]:
pd.isnull(df['words'])
Out[46]:
0 False
1 False
2 False
3 False
4 False
5 False
6 True
7 False
8 False
Name: words, dtype: bool
the question was about the if:
Use with AIfor i in range(150,160):
if (pd.isnull(df['Text'][i])):
print('is null')
or count all NaN:
Use with AIcnt=0
for i in range(len(dframe)):
if (pd.isnull(dframe['Description'][i])):
cnt+=1
Methods for this already exist, particularly because of the weird properties of NaNs. One of them can be found in the math library, math.isnan and numpy already has a method implemented for this as well, numpy.isnan. They both deal with all three kinds of NaNs shown in your code (but the numpy version is vectorized):
import math
import numpy as np
list(map(math.isnan, [float("nan"), math.nan, np.nan]))
# [True, True, True]
np.isnan([float("nan"), math.nan, np.nan])
# array([ True, True, True])
If you are using pandas, pandas.isna is also interesting because it considers None and pandas.NaT (Not a Time) as missing values as well:
import pandas as pd
pd.isna([float("nan"), math.nan, np.nan, None, pd.NaT])
# array([ True, True, True, True, True])
Your code has a potential flaw, it does not return True only for NaNs (although it does return True for all NaNs), but for all objects which are not equal to themselves. In particular, it will think any instance of this class is NaN (probably true, but not necessarily):
class A:
def __equal__(self, other):
return False
IMHO your function is correct and readable as it is.
If I see something like x != x in any language my mind jumps immediately to NaN,
regardless of the type.
So if you have something like a missing string or date or whatever, this test still makes sense.
Regarding the report from sonarsource. It is always helpful to read the rationale behind a rule. If you look here there are nice examples why this rule generally makes sense. I don't think this applies in your case and you can ignore it.
Some nitpicks:
- I would directly return without creating a temporary. The same is true for printing.
- There are two spaces after is_nan =
- Your functions are not in a class and are really functions. Even if they were methods it would be very redundant to add it. Why don't you just call it
is_nan?
import math
import numpy as np
v1 = float('nan')
v2 = math.nan
v3 = np.nan
def is_nan(value) -> bool:
return value != value
print(is_nan(v1))
print(is_nan(v2))
print(is_nan(v3))
You could add your variables which you'd like to interpret as NaN to na_values argument of the pd.read_csv:
df = pd.read_csv('your_file.csv', na_values=['NAN'])
Also you could find some information in that answer.
All default NA values from na-values:
The default NaN recognized values are
['-1.#IND', '1.#QNAN', '1.#IND', '-1.#QNAN', '#N/A','N/A', 'NA', '#NA', 'NULL', 'NaN', '-NaN', 'nan', '-nan']. Although a 0-length string '' is not included in the default NaN values list, it is still treated as a missing value.
What kind of error? It seems to be working fine. I tried Python2 and 3.
>>> float("NAN")
nan
>>> float("NaN")
nan
>>> float("nan")
nan
Perhaps you are trying to convert to int? The int type doesn't allow a "NaN" value.