As you can see, the AND operator drops every row in which at least one value equals -1. On the other hand, the OR operator requires both values to be equal to -1 to drop them.
That's right. Remember that you're writing the condition in terms of what you want to keep, not in terms of what you want to drop. For df1:
df1 = df[(df.a != -1) & (df.b != -1)]
You're saying "keep the rows in which df.a isn't -1 and df.b isn't -1", which is the same as dropping every row in which at least one value is -1.
For df2:
df2 = df[(df.a != -1) | (df.b != -1)]
You're saying "keep the rows in which either df.a or df.b is not -1", which is the same as dropping rows where both values are -1.
PS: chained access like df['a'][1] = -1 can get you into trouble. It's better to get into the habit of using .loc and .iloc.
As you can see, the AND operator drops every row in which at least one value equals -1. On the other hand, the OR operator requires both values to be equal to -1 to drop them.
That's right. Remember that you're writing the condition in terms of what you want to keep, not in terms of what you want to drop. For df1:
df1 = df[(df.a != -1) & (df.b != -1)]
You're saying "keep the rows in which df.a isn't -1 and df.b isn't -1", which is the same as dropping every row in which at least one value is -1.
For df2:
df2 = df[(df.a != -1) | (df.b != -1)]
You're saying "keep the rows in which either df.a or df.b is not -1", which is the same as dropping rows where both values are -1.
PS: chained access like df['a'][1] = -1 can get you into trouble. It's better to get into the habit of using .loc and .iloc.
Late answer, but you can also use query(), e.g. :
df_filtered = df.query('a == 4 & b != 2')
How to "pass through" multiple conditions in a pandas dataframe with query?
python - Pandas: np.where with multiple conditions on dataframes - Stack Overflow
Pandas - Filter based on multiple conditions
python - Pandas dataframe numpy where multiple conditions - Stack Overflow
Users can use the where or query function with pandas dataframes to select rows/columns of the dataframe that match certain conditions, e.g.
http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.query.html
>>> from numpy.random import randn
>>> from pandas import DataFrame
>>> df = DataFrame(randn(10, 2), columns=list('ab'))
>>> df.query('a > b')In my case, it may be better to think of a simple conditional, all rows satisfying a==1
df.query('a==1')Let's say I had a numpy array/Python list of values, and I would like to do an "OR" query for each item in the list.
list1 = [10, 20, 50]
# the query
df.query('a==10 | a==20 | a==50')Is this possible? The idea would be I would write a function whereby users input a list of values to query, and it performs an OR query for each.
For where, the idea is similar:
temperatures = [80, 90, 100]
# reads in temperatures
# performs this query:
rows = df.where('(temperature == 80) | (temperature == 90) | (temperature == 100)')It is not clear to me what you exactly want to do when a y element is equal to zero... anyway the key point in this answer is "use np.logical_{and,not,or,xor} functions".
I think that the following, albeit formulated differently from your example, does what you want, but if I'm wrong you should be able to combine different tests to achieve what you want,
x = np.where(np.logical_or(x*y>0, y==0), x, 0)
Similar to the post by @gboffi, but more centralized for your original request according to my understanding try:
x = np.where(np.logical_and((x*y) < 0, x != 0))
or
x = np.where(((x*y) < 0) & (x != 0)))
Hi,
I have a csv file with approx. 100 columns and I want to filter rows if two of the columns are set to a value of X and the other columns are blank / Nan values. In order to make the code more readable, I would like to specify the column names in a list and then use the variable name within the Pandas query e.g. something like the following:
my_file=/home/test.csv my_df=pd.read_csv(my_file) control_fields=['Is_Active','Is_Valid'] data_fields=['Age','DOB','Country','City']
As a starting point, I have tried the following but this isn't filtering the data at all:
my_new_df=my_df[my_df['control_fields']==1]
Can someone please explain why the above isn't working and also advise if there is a better way of achieving my requirement?
Thanks!