x = ['1', '2', '4', 'c'], so x[1]=='2', which makes the expression (x[0] != "1" and x[1] != "2" and x[2] != "3") be evaluated as False.
When conditions are joined by and, they return True only if all conditions are True, and if they are joined by or, they return True when the first among them is evaluated to be True.
x = ['1', '2', '4', 'c'], so x[1]=='2', which makes the expression (x[0] != "1" and x[1] != "2" and x[2] != "3") be evaluated as False.
When conditions are joined by and, they return True only if all conditions are True, and if they are joined by or, they return True when the first among them is evaluated to be True.
['1', '2', '4', 'c']
Fails for condition
x[0] != "1"
as well as
x[1] != "2"
Instead of using or, I believe the more natural and readable way is:
lambda x: (x[0], x[1], x[2]) != ('1','2','3')
Out of curiosity, I compared three methods of, er... comparing, and the results were as expected: slicing lists was the slowest, using tuples was faster, and using boolean operators was the fastest. More precisely, the three approaches compared were
list_slice_compare = lambda x: x[:3] != [1,2,3]
tuple_compare = lambda x: (x[0],x[1],x[2]) != (1,2,3)
bool_op_compare = lambda x: x[0]!= 1 or x[1] != 2 or x[2]!= 3
And the results, respectively:
In [30]: timeit.Timer(setup="import timeit,random; rand_list = [random.randint(1,9) for _ in range(4)]; list_slice_compare = lambda x: x[:3] != [1,2,3]", stmt="list_slice_compare(rand_list)").repeat()
Out[30]: [0.3207617177499742, 0.3230015148823213, 0.31987868894918847]
In [31]: timeit.Timer(setup="import timeit,random; rand_list = [random.randint(1,9) for _ in range(4)]; tuple_compare = lambda x: (x[0],x[1],x[2]) != (1,2,3)", stmt="tuple_compare(rand_list)").repeat()
Out[31]: [0.2399928924012329, 0.23692036176475995, 0.2369164465619633]
In [32]: timeit.Timer(setup="import timeit,random; rand_list = [random.randint(1,9) for _ in range(4)]; bool_op_compare = lambda x: x[0]!= 1 or x[1] != 2 or x[2]!= 3", stmt="bool_op_compare(rand_list)").repeat()
Out[32]: [0.144389363900018, 0.1452672728203197, 0.1431527621755322]
pandas - Filtering multiple conditions from a Dataframe in Python - Stack Overflow
python - Pandas: Filtering multiple conditions - Stack Overflow
conditional statements - Python : How to filter multiple conditions - Stack Overflow
Python - Filter function with multiple conditions - Stack Overflow
Use () because operator precedence:
temp2 = df[~df["Def"] & (df["days since"] > 7) & (df["bin"] == 3)]
Alternatively, create conditions on separate rows:
cond1 = df["bin"] == 3
cond2 = df["days since"] > 7
cond3 = ~df["Def"]
temp2 = df[cond1 & cond2 & cond3]
Sample:
df = pd.DataFrame({'Def':[True] *2 + [False]*4,
'days since':[7,8,9,14,2,13],
'bin':[1,3,5,3,3,3]})
print (df)
Def bin days since
0 True 1 7
1 True 3 8
2 False 5 9
3 False 3 14
4 False 3 2
5 False 3 13
temp2 = df[~df["Def"] & (df["days since"] > 7) & (df["bin"] == 3)]
print (temp2)
Def days since bin
3 False 14 3
5 False 13 3
Another idea is use functions like Series.lt
Series.gt
Series.le
Series.ge
Series.ne:
temp2 = df[~df["Def"] & df["days since"].gt(7) & df["bin"].eq(3)]
print (temp2)
Def days since bin
3 False 14 3
5 False 13 3
OR
df_train[(df_train["fold"]==1) | (df_train["fold"]==2)]
AND
df_train[(df_train["fold"]==1) & (df_train["fold"]==2)]
The logic behind the code is wrong.
If you do a_list['type2'] != unwanted_type2 is like a_list['type2'] !={'Apple','Banana','Orange','Melon'}, and you are comparing a value to the entire unwated_type2.
To resolve it, you can use not in in the condition:
B = []
wanted_type1 = 'A'
unwanted_type2 = {'Apple','Banana','Orange','Melon'}
unwanted_type3 = {'stage','books','films','music'}
a_list = some_variable['response']['results']
for list in a_list:
if (a_list['type1'] == 'A') and (a_list['type2'] not in unwanted_type2) and (a_list['typeb'] not in unwanted_type3):
B.append(list['type4'])
With this method you are looking at
Problem is that you are comparing "string" and list(or iterable)
Try changing code to something below:
unwanted_type2 = ['Apple','Banana','Orange','Melon']
if "Apple" in unwanted_type2:
print("Condition works!")
if "Raspberry" not in unwanted_type2:
print("Negative condition works!")
You need:
fil_1 = test['col_a'].isin(['abc','def','ghi'])
fil_2 = test['col_b'].isin(['yes'])
fil_3 = test['col_c'].isin(['a'])
or
test.isin({'col_a': ['abc','def','ghi'],
'col_b': ['yes'],
'col_c' :['a']}).all(axis = 1)
df_filtered = test[fil_1 & fil_2 & fil_3]
print(df_filtered)
col_a col_b col_c
0 abc yes a
2 abc yes a
4 def yes a
6 def yes a
8 ghi yes a
10 ghi yes a
or logic |
fil = test.isin({'col_a': ['abc','def','ghi'],'col_b': ['yes'],'col_c' :['a']})
df_filtered = df[fil]
print(df_filtered)
col_a col_b col_c
0 abc yes a
1 abc NaN NaN
2 abc yes a
3 def NaN NaN
4 def yes a
5 def NaN NaN
6 def yes a
7 def NaN NaN
8 ghi yes a
9 ghi NaN NaN
10 ghi yes a
Now if we also use DataFrame.all:
df_filtered = df[fil.all(axis = 1)]
print(df_filtered)
col_a col_b col_c
0 abc yes a
2 abc yes a
4 def yes a
6 def yes a
8 ghi yes a
10 ghi yes a
Detail
print(fil)
col_a col_b col_c
0 True True True
1 True False False
2 True True True
3 True False False
4 True True True
5 True False False
6 True True True
7 True False False
8 True True True
9 True False False
10 True True True
print(test.isin({'col_a': ['abc','def','ghi']}))
col_a col_b col_c
0 True False False
1 True False False
2 True False False
3 True False False
4 True False False
5 True False False
6 True False False
7 True False False
8 True False False
9 True False False
10 True False False
this return False in columns differences than col_a
so you got NaN values because you were using &
Here's the one-liner solution,
test[test.col_a.isin(['abc','def','ghi']) & test.col_b.isin(['yes']) & test.col_c.isin(['a'])]
Pandas (and numpy) allow for boolean indexing, which will be much more efficient:
In [11]: df.loc[df['col1'] >= 1, 'col1']
Out[11]:
1 1
2 2
Name: col1
In [12]: df[df['col1'] >= 1]
Out[12]:
col1 col2
1 1 11
2 2 12
In [13]: df[(df['col1'] >= 1) & (df['col1'] <=1 )]
Out[13]:
col1 col2
1 1 11
If you want to write helper functions for this, consider something along these lines:
In [14]: def b(x, col, op, n):
return op(x[col],n)
In [15]: def f(x, *b):
return x[(np.logical_and(*b))]
In [16]: b1 = b(df, 'col1', ge, 1)
In [17]: b2 = b(df, 'col1', le, 1)
In [18]: f(df, b1, b2)
Out[18]:
col1 col2
1 1 11
Update: pandas 0.13 has a query method for these kind of use cases, assuming column names are valid identifiers the following works (and can be more efficient for large frames as it uses numexpr behind the scenes):
In [21]: df.query('col1 <= 1 & 1 <= col1')
Out[21]:
col1 col2
1 1 11
Chaining conditions creates long lines, which are discouraged by PEP8.
Using the .query method forces to use strings, which is powerful but unpythonic and not very dynamic.
Once each of the filters is in place, one approach could be:
import numpy as np
import functools
def conjunction(*conditions):
return functools.reduce(np.logical_and, conditions)
c_1 = data.col1 == True
c_2 = data.col2 < 64
c_3 = data.col3 != 4
data_filtered = data[conjunction(c_1,c_2,c_3)]
np.logical operates on and is fast, but does not take more than two arguments, which is handled by functools.reduce.
Note that this still has some redundancies:
- Shortcutting does not happen on a global level
- Each of the individual conditions runs on the whole initial data
Still, I expect this to be efficient enough for many applications and it is very readable. You can also make a disjunction (wherein only one of the conditions needs to be true) by using np.logical_or instead:
import numpy as np
import functools
def disjunction(*conditions):
return functools.reduce(np.logical_or, conditions)
c_1 = data.col1 == True
c_2 = data.col2 < 64
c_3 = data.col3 != 4
data_filtered = data[disjunction(c_1,c_2,c_3)]
Question is to find all the prime numbers between two given numbers.
i made a 'list' of numbers between the two given number(let the 2 numbers be 1 and 100).
then i did this
l1=list(filter(lambda x:x!=2 and x%2,l))
where l is my list with all numbers.
output i got [1,3,5,6.....99]
shouldn't my output be [1,2,3....99]
Hi,
I have a csv file with approx. 100 columns and I want to filter rows if two of the columns are set to a value of X and the other columns are blank / Nan values. In order to make the code more readable, I would like to specify the column names in a list and then use the variable name within the Pandas query e.g. something like the following:
my_file=/home/test.csv my_df=pd.read_csv(my_file) control_fields=['Is_Active','Is_Valid'] data_fields=['Age','DOB','Country','City']
As a starting point, I have tried the following but this isn't filtering the data at all:
my_new_df=my_df[my_df['control_fields']==1]
Can someone please explain why the above isn't working and also advise if there is a better way of achieving my requirement?
Thanks!
You can filter your list:
list_of_stuff = [
"aus-airport-1",
"aus-airport-2",
"us-airport-1",
"us-airport-2",
"aus-ship-1",
"us-ship-99",
"nz-airport-1"
]
is_airport = lambda x: "-airport-" in x
is_ship = lambda x: "-ship-" in x
airports_excluding_nz = lambda x: is_airport(x) and not x.startswith("nz-")
airports_in_aus = lambda x: is_airport(x) and x.startswith("nz-")
ships = lambda x: is_ship(x)
print ("all regions excluding nz:" ,
", ".join( filter(airports_excluding_nz , list_of_stuff) ) )
print ("all regions in aus:",
", ".join( filter(airports_in_aus, list_of_stuff) ) )
print ("all ships:",
", ".join( filter(ships, list_of_stuff) ) )
Check results:
all regions excluding nz aus-airport-1, aus-airport-2, us-airport-1, us-airport-2
all regions in aus nz-airport-1
all ships aus-ship-1, us-ship-99
You can use three straight list-comprehensions:
lst = ["aus-airport-1","aus-airport-2","us-airport-1","us-airport-2","aus-ship-1","us-ship-99","nz-airport-1"]
splits = list(map(lambda x: x.split('-'), lst))
lst1 = [x for x in splits if x[1] == 'airport' and x[0] != 'nz']
print(f'All airports excluding nz: {lst1}')
lst2 = [x for x in splits if x[1] == 'airport' and x[0] == 'aus']
print(f'All airports in aus: {lst2}')
lst3 = [x for x in splits if x[1] == 'ship']
print(f'All ships: {lst3}')