jwilner's response is spot on. I was exploring to see if there's a faster option, since in my experience, summing flat arrays is (strangely) faster than counting. This code seems faster:

df.isnull().values.any()

import numpy as np
import pandas as pd
import perfplot


def setup(n):
    df = pd.DataFrame(np.random.randn(n))
    df[df > 0.9] = np.nan
    return df


def isnull_any(df):
    return df.isnull().any()


def isnull_values_sum(df):
    return df.isnull().values.sum() > 0


def isnull_sum(df):
    return df.isnull().sum() > 0


def isnull_values_any(df):
    return df.isnull().values.any()


perfplot.save(
    "out.png",
    setup=setup,
    kernels=[isnull_any, isnull_values_sum, isnull_sum, isnull_values_any],
    n_range=[2 ** k for k in range(25)],
)

df.isnull().sum().sum() is a bit slower, but of course, has additional information -- the number of NaNs.

So python has the short-circuiting logic operators not, and, or. These have a very specific meaning in python and cannot be overridden (not must return a bool and a and/or b always returns either a or b or throws an error.

However, python also has over-loadable boolean operators ~ (not), & (and), | (or) and ^ (xor).

You may recognise these as the int bitwise operators, but Numpy (and therefore pandas) use these to do array / series boolean operations.

For example

b = np.array([True, False, True]) & np.array([True, False, False])
# b --> [True False False]
b = ~b 
# b --> [False True True]

Hence what you want is

df = df[~df['my_col'].isnull()]

I agree with PEP8, don't do == False.

Use .:

a = df_data.query('a.isnull()', engine='python')
print (a)
    a
2 NaN

b = df_data.query('a.notnull()', engine='python')
print (b)
      a
0   1.0
1  20.0
3  40.0
4  50.0

You can use also logic NaN != NaN:

a = df_data.query('a != a')
print (a)
    a
 2 NaN

b = df_data.query('a == a')
print (b)
      a
0   1.0
1  20.0
3  40.0
4  50.0

Using pandas, you should avoid loop. Use mask filtering and slicing to fill your flag column. In order to detect null values, use .isnull() directly on pandas dataframe or series (when you select a column), not on a value as you did. Then use .fillna() if you want to replace null values with something else.

Based on your code (but not sure it will works, it could be helpfull you share some input data and expected output), the solution may look as follow.

First create empty column as you did:

data['Flags'] = None

Then fill this columns based on condition on "Temperature phase" column (using fillna(0) to replace all null values by 0 allow you to only test if values are <= 0, this replacement is not applied on the final dataframe):

data.loc[data['Temperature phase'].fillna(0) <= 0, "Flags"] = 1
data.loc[data['Temperature phase'] > 200, "Flags"] = 2

And now replace Temperature phase values.

For the values equal to 0 or null, you seems to have choosen to replace them with the previous value in dataframe. You maybe could achieve this part using this.

data.loc[data['Temperature phase'].isnull(), 'Temperature phase'] = data['Temperature phase'].shift().loc[data.loc[data['Temperature phase'].isnull()].index]

First, this command use .shift() to shift all values in column Temperature phase by one, then filtering rows where Temperature phase is null and replace values by corresponding index in shifted Temperature phase values.

Finaly, replace other Temperature phase values:

data.loc[data['Temperature phase'] < 0, "Temperature phase"] = 0
data.loc[data['Temperature phase'] > 200, "Temperature phase"] = 130

You don't need flag index so on as the Flag is directly fill in the final dataframe.

The problem is that pd.notnull(['foo', 'bar']) operates elementwise and returns array([ True, True], dtype=bool). Your if condition trys to convert that to a boolean, and that's when you get the exception.

To fix it, you could simply wrap the isnull statement with np.all:

df[['A','C']].apply(lambda x: my_func(x) if(np.all(pd.notnull(x[1]))) else x, axis = 1)

Now you'll see that np.all(pd.notnull(['foo', 'bar'])) is indeed True.

you can use .isna() method:

In [48]: df[df[2].isna()]
Out[48]:
   0  1   2
1  3  4 NaN