I think using functions here is unnecessary. It is better and mainly faster to use boolean indexing:
m = (df['Name'] == 'Alisa') & (df['Age'] > 24)
print(m)
0 True
1 False
2 False
3 False
4 False
5 False
6 True
7 False
8 False
9 False
10 False
11 False
dtype: bool
#invert mask by ~
df1 = df[~m]
For more complicated filtering, you could use a function which must return a boolean value:
def filter_fn(row):
if row['Name'] == 'Alisa' and row['Age'] > 24:
return False
else:
return True
df = pd.DataFrame(d, columns=['Name', 'Age', 'Score'])
m = df.apply(filter_fn, axis=1)
print(m)
0 False
1 True
2 True
3 True
4 True
5 True
6 False
7 True
8 True
9 True
10 True
11 True
dtype: bool
df1 = df[m]
Answer from jezrael on Stack OverflowI think using functions here is unnecessary. It is better and mainly faster to use boolean indexing:
m = (df['Name'] == 'Alisa') & (df['Age'] > 24)
print(m)
0 True
1 False
2 False
3 False
4 False
5 False
6 True
7 False
8 False
9 False
10 False
11 False
dtype: bool
#invert mask by ~
df1 = df[~m]
For more complicated filtering, you could use a function which must return a boolean value:
def filter_fn(row):
if row['Name'] == 'Alisa' and row['Age'] > 24:
return False
else:
return True
df = pd.DataFrame(d, columns=['Name', 'Age', 'Score'])
m = df.apply(filter_fn, axis=1)
print(m)
0 False
1 True
2 True
3 True
4 True
5 True
6 False
7 True
8 True
9 True
10 True
11 True
dtype: bool
df1 = df[m]
A very readable way to filter dataframes is query.
df.query("not (Name == 'Alisa' and Age > 24)")
# or pass the negation from the beginning (by de Morgan's laws)
df.query("Name != 'Alisa' or Age <= 24")
Another way is to pass the complicated function to loc to filter.
df.loc[lambda x: ~((x['Name'] == 'Alisa') & (x['Age'] > 24))]

python - pandas: complex filter on rows of DataFrame - Stack Overflow
Python, Pandas: Filter rows of data frame based on function - Stack Overflow
Pandas how do I 'apply' and filter at the same time?
Polars: How to filter columns by date range?
You can do this using DataFrame.apply, which applies a function along a given axis,
In [3]: df = pandas.DataFrame(np.random.randn(5, 3), columns=['a', 'b', 'c'])
In [4]: df
Out[4]:
a b c
0 -0.001968 -1.877945 -1.515674
1 -0.540628 0.793913 -0.983315
2 -1.313574 1.946410 0.826350
3 0.015763 -0.267860 -2.228350
4 0.563111 1.195459 0.343168
In [6]: df[df.apply(lambda x: x['b'] > x['c'], axis=1)]
Out[6]:
a b c
1 -0.540628 0.793913 -0.983315
2 -1.313574 1.946410 0.826350
3 0.015763 -0.267860 -2.228350
4 0.563111 1.195459 0.343168
Suppose I had a DataFrame as follows:
In [39]: df
Out[39]:
mass1 mass2 velocity
0 1.461711 -0.404452 0.722502
1 -2.169377 1.131037 0.232047
2 0.009450 -0.868753 0.598470
3 0.602463 0.299249 0.474564
4 -0.675339 -0.816702 0.799289
I can use sin and DataFrame.prod to create a boolean mask:
In [40]: mask = (np.sin(df.velocity) / df.ix[:, 0:2].prod(axis=1)) > 0
In [41]: mask
Out[41]:
0 False
1 False
2 False
3 True
4 True
Then use the mask to select from the DataFrame:
In [42]: df[mask]
Out[42]:
mass1 mass2 velocity
3 0.602463 0.299249 0.474564
4 -0.675339 -0.816702 0.799289
I think you can just filter based in 13th symbol of your string:
import pandas as pd
# Enter some data. We want to filter out all rows where the number at pos 13,14 > 9
df = pd.DataFrame({
'ID': ['ABCD-3Z-A93Z-01A-11R-A37O-07',
'ABCD-6D-AA2E-11A-11R-A37O-07',
'ABCD-6D-AA2E-01A-11R-A37O-07',
'ABCD-A3-3307-01A-01R-0864-07',
'ABCD-6D-AA2E-01A-11R-A37O-07',
'ABCD-6D-AA2E-10A-11R-A37O-07',
'ABCD-6D-AA2E-09A-11R-A37O-07'],
'year': [2012, 2012, 2013, 2014, 2014, 2017, 2015]
})
# convert to df
df['KeepRow'] = df['ID'].apply(lambda x: x[13] == '0')
or simply:
df[df['ID'].apply(lambda x: x[13] == '0')]
Use indexing with str for values by positions, then convert to float and filter by boolean indexing:
df = df[df['ID'].str[13:15].astype(float) <=9]
print(df)
ID year
0 ABCD-3Z-A93Z-01A-11R-A37O-07 2012
2 ABCD-6D-AA2E-01A-11R-A37O-07 2013
3 ABCD-A3-3307-01A-01R-0864-07 2014
4 ABCD-6D-AA2E-01A-11R-A37O-07 2014
6 ABCD-6D-AA2E-09A-11R-A37O-07 2015
Detail:
print(df['ID'].str[13:15])
0 01
1 11
2 01
3 01
4 01
5 10
6 09
Name: ID, dtype: object