Use math.isnan:
>>> import math
>>> x = float('nan')
>>> math.isnan(x)
True
Answer from gimel on Stack OverflowUse math.isnan:
>>> import math
>>> x = float('nan')
>>> math.isnan(x)
True
The usual way to test for a NaN is to see if it's equal to itself:
def isNaN(num):
return num != num
python - How to understand float('nan')? What is the use of float? - Stack Overflow
constants - Assigning a variable NaN in python without numpy - Stack Overflow
python - How to set a number to NaN or infinity? - Stack Overflow
math.isnan() vs = float('nan')
A float is a 64-bit* IEEE-754 Floating-Point Number.
Most values that can be represented by floats, we can write as literals (e.g. 1.5 or 52.0). However, the values for Infinity (inf) and "Not-a-Number" (NaN), which the IEEE-754 standard requires, can only be reliably assigned by converting those strings into floats. That's what float('nan') does - it creates a float whose value is NaN.
The article you read is misleading. According to the IEEE-754 standard, the value NaN works differently from any other float value - it will always return false when used in any comparison with another number, unless that comparison is !=, in which case it will always return true. This is even the case when comparing it to itself:
Use with AI>>> var = float('nan')
>>> var != var
True
as such, using NaN in comparisons is not very useful.
Instead, to check if a variable contains the value NaN, do
Use with AIimport math
...
if math.isnan(some_variable):
...
or just
Use with AIif some_variable == some_variable:
...
since that will always be true if some_variable is not NaN, and will always be false if it is NaN.
The article is probably guiding you to check whether the values in a column of the dataframe are NaN and use different behavior accordingly.
*in python, at least - in most languages float is 32 bits and double is the 64-bit equivalent
NaN (Not A Number) is classically a way to identify missing values. It is a float type object and can be defined using first('nan') the fake way that 1.23 can be defined using float('1.23'). You can also define infinite with float('inf').
Some modules directly define a NaN object, which is the case of numpy:
Use with AIfrom numpy import nan
If you are working with integers, strings, etc. in pandas, you can also use the type-aware NA with pandas.NA/pd.NA.
Further references:
- pandas: working with missing data
- pandas: nullable integer data type
Regarding your question, it is difficult to answer without reading the article, but what you described is incorrect.
print(df == float('nan')) will always return False as float('nan')==float('nan') is False.
print(df = float('nan')) will raise a SyntaxError.
The correct way to check for NaN values is isna
Use with AIdf.isna()
Which returns one True or False per cell.
To get an aggregated value, you need to combine for example with any
Use with AI# is there at least one NaN in the whole DataFrame?
df.isna().any().any()
Yes -- use math.nan.
>>> from math import nan
>>> print(nan)
nan
>>> print(nan + 2)
nan
>>> nan == nan
False
>>> import math
>>> math.isnan(nan)
True
Before Python 3.5, one could use float("nan") (case insensitive).
Note that checking to see if two things that are NaN are equal to one another will always return false. This is in part because two things that are "not a number" cannot (strictly speaking) be said to be equal to one another -- see What is the rationale for all comparisons returning false for IEEE754 NaN values? for more details and information.
Instead, use math.isnan(...) if you need to determine if a value is NaN or not.
Furthermore, the exact semantics of the == operation on NaN value may cause subtle issues when trying to store NaN inside container types such as list or dict (or when using custom container types). See Checking for NaN presence in a container for more details.
You can also construct NaN numbers using Python's decimal module:
>>> from decimal import Decimal
>>> b = Decimal('nan')
>>> print(b)
NaN
>>> print(repr(b))
Decimal('NaN')
>>>
>>> Decimal(float('nan'))
Decimal('NaN')
>>>
>>> import math
>>> math.isnan(b)
True
math.isnan(...) will also work with Decimal objects.
However, you cannot construct NaN numbers in Python's fractions module:
>>> from fractions import Fraction
>>> Fraction('nan')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python35\lib\fractions.py", line 146, in __new__
numerator)
ValueError: Invalid literal for Fraction: 'nan'
>>>
>>> Fraction(float('nan'))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python35\lib\fractions.py", line 130, in __new__
value = Fraction.from_float(numerator)
File "C:\Python35\lib\fractions.py", line 214, in from_float
raise ValueError("Cannot convert %r to %s." % (f, cls.__name__))
ValueError: Cannot convert nan to Fraction.
Incidentally, you can also do float('Inf'), Decimal('Inf'), or math.inf (3.5+) to assign infinite numbers. (And also see math.isinf(...))
However doing Fraction('Inf') or Fraction(float('inf')) isn't permitted and will throw an exception, just like NaN.
If you want a quick and easy way to check if a number is neither NaN nor infinite, you can use math.isfinite(...) as of Python 3.2+.
If you want to do similar checks with complex numbers, the cmath module contains a similar set of functions and constants as the math module:
cmath.isnan(...)cmath.isinf(...)cmath.isfinite(...)(Python 3.2+)cmath.nan(Python 3.6+; equivalent tocomplex(float('nan'), 0.0))cmath.nanj(Python 3.6+; equivalent tocomplex(0.0, float('nan')))cmath.inf(Python 3.6+; equivalent tocomplex(float('inf'), 0.0))cmath.infj(Python 3.6+; equivalent tocomplex(0.0, float('inf')))
nan = float('nan')
And now you have the constant, nan.
You can similarly create NaN values for decimal.Decimal.:
dnan = Decimal('nan')
Cast from string using float():
>>> float('NaN')
nan
>>> float('Inf')
inf
>>> -float('Inf')
-inf
>>> float('Inf') == float('Inf')
True
>>> float('Inf') == 1
False
Yes, you can use numpy for that.
import numpy as np
a = arange(3,dtype=float)
a[0] = np.nan
a[1] = np.inf
a[2] = -np.inf
a # is now [nan,inf,-inf]
np.isnan(a[0]) # True
np.isinf(a[1]) # True
np.isinf(a[2]) # True
I have a variable that I am setting as a nan via = float('nan') under certain conditions, and when I check later on via math.isnan() whether this in fact a nan, I don't True. However, when I use == float('nan'), I do get "True" as result.
What could be the cause of this?
Why not just wrap whatever you pass to math.isnan with another float conversion? If it was already a float (i.e. float('nan')) you just made a "redundant" call:
import math
def is_nan(value):
return math.isnan(float(value))
And this seems to give your expected results:
>>> is_nan(float('nan'))
True
>>> is_nan('nan')
True
>>> is_nan(np.nan)
True
>>> is_nan(5)
False
>>> is_nan('5')
False
This will still raise a ValueError for non-numeric (except 'nan') strings. If that's a problem, you can wrap with try/except. As long as the float conversion worked, there is no reason for isnan to fail. So we are basically catching non-numeric strings that my fail the float conversion:
def is_nan(value):
try:
return math.isnan(float(value))
except ValueError:
return False
Any non-numeric string is surely not a NaN value so return False.
It's very pedestrian, and a bit ugly, but why not just do the following?
import math
import numpy as np
if math.isnan(x) or x in ['nan', np.nan, ... ]:
# Do something
pass
I want to recommend a Yoda expression but haven't quite worked it out yet.
If you want to sweep everything under the carpet put it in a lambda or function.
Following on from https://stackoverflow.com/a/64090763/1021819, you can try to get the iterator to evaluate any in a lazy fashion. The problem then is that if none of the first conditions evaluates to True then the math.isnan() call is executed and can still throw the TypeError. If you evaluate lazily you can guard the math.isnan() call with a type check against str:
fn_list_to_check=[
lambda x: x in ['nan', np.nan, ... ],
lambda x: not isinstance(x, str),
lambda x: math.isnan(x)
]
if any(f(x) for f in fn_list_to_check):
# Do something
pass
Note the absence of square list brackets in the any i.e. any() not any([]) (who knew?).
I think it's quite brilliant but equally as ugly - choose your poison.
For the second part of the question (why float('nan') != float('nan')), see
What is the rationale for all comparisons returning false for IEEE754 NaN values?