Setup
Borrowed @MaxU's df
df = pd.DataFrame([
[1, 2, 3],
[4, None, 6],
[None, 7, 8],
[9, 10, 11]
], dtype=object)
Solution
You can just use pd.DataFrame.dropna as is
df.dropna()
0 1 2
0 1 2 3
3 9 10 11
Supposing you have None strings like in this df
df = pd.DataFrame([
[1, 2, 3],
[4, 'None', 6],
['None', 7, 8],
[9, 10, 11]
], dtype=object)
Then combine dropna with mask
df.mask(df.eq('None')).dropna()
0 1 2
0 1 2 3
3 9 10 11
You can ensure that the entire dataframe is object when you compare with.
df.mask(df.astype(object).eq('None')).dropna()
0 1 2
0 1 2 3
3 9 10 11
Answer from piRSquared on Stack OverflowSetup
Borrowed @MaxU's df
df = pd.DataFrame([
[1, 2, 3],
[4, None, 6],
[None, 7, 8],
[9, 10, 11]
], dtype=object)
Solution
You can just use pd.DataFrame.dropna as is
df.dropna()
0 1 2
0 1 2 3
3 9 10 11
Supposing you have None strings like in this df
df = pd.DataFrame([
[1, 2, 3],
[4, 'None', 6],
['None', 7, 8],
[9, 10, 11]
], dtype=object)
Then combine dropna with mask
df.mask(df.eq('None')).dropna()
0 1 2
0 1 2 3
3 9 10 11
You can ensure that the entire dataframe is object when you compare with.
df.mask(df.astype(object).eq('None')).dropna()
0 1 2
0 1 2 3
3 9 10 11
Thanks for all your help. In the end I was able to get
df = df.replace(to_replace='None', value=np.nan).dropna()
to work. I'm not sure why your suggestions didn't work for me.
You can nest list comprehensions:
df['Mar'] = [[x for x in inner_list if x is not None] for inner_list in df['Mar']]
You can also use filter to filter out None values
df['Mar'] = [list(filter(lambda x: x is None, inner_list)) for inner_list in df['Mar']]
@user3080953 pretty much solved this, but I wanted to put a complete answer for posterity's sake:
df['New'] = [[[x for x in inner_list if x is not None]] for inner_list in df['Mar']]
This creates a new column that is "None"-free. I can then delete df['Mar']
Thanks all!
Use:
#DataFrame from sample data
df_out = pd.DataFrame(df_out)
#filter columns names by list and test if NaN or None at least in one row
m = df_out[['aaa','bbb']].isna().any(axis=1)
#OR test both columns separately
m = df_out['aaa'].isna() | df_out['bbb'].isna()
#filter matched and not matched rows
df1 = df_out[m].reset_index(drop=True)
df2 = df_out[~m].reset_index(drop=True)
print (df1)
name aaa bbb
0 Mick None None
1 Ivan-Peter 1 None
print (df2)
name aaa bbb
0 Ivan A C
1 Juli 1 P
Another idea with DataFrame.dropna and filter indices not exist in df2:
df2 = df_out.dropna()
df1 = df_out.loc[df_out.index.difference(df2.index)].reset_index(drop=True)
df2 = df2.reset_index(drop=True)
First of all one needs to convert df_out to a dataframe with pandas.DataFrame as follows
df_out = pd.DataFrame(df_out)
[Out]:
name aaa bbb
0 Mick None None
1 Ivan A C
2 Ivan-Peter 1 None
3 Juli 1 P
Then one can use, for both cases, pandas.Series.notnull.
With values, where we have None in columns aaa and/or bbb, named filter_nulls in my code
df1 = df_out[~df_out['aaa'].notnull() | ~df_out['bbb'].notnull()]
[Out]:
name aaa bbb
0 Mick None None
2 Ivan-Peter 1 None
Where we do not have None at all. df_out in my code.
df2 = df_out[df_out['aaa'].notnull() & df_out['bbb'].notnull()]
[Out]:
name aaa bbb
1 Ivan A C
3 Juli 1 P
Notes:
If needed one can use
pandas.DataFrame.reset_indexto get the followingdf_new = df_out[~df_out['aaa'].notnull() | ~df_out['bbb'].notnull()].reset_index(drop=True) [Out]: name aaa bbb 0 Mick None None 1 Ivan-Peter 1 None
I'm trying to remove rows from a dataframe where a specific column (license) is None. The code I'm trying is below
df_autos = pd.read_sql_query('SELECT * FROM "autos"', con=engine)
print('df_autos ',df_autos,len(df_autos))
df_autos = df_autos[df_autos['license'].apply(lambda x: x is not None)]
print('df_autos after removal ',df_autos,len(df_autos))but when I look in the terminal at the second print statement the length is the same. The df_autos remains unchanged, what I want to see is the length get reduced for each row where None is the value for the license column. Any ideas...