pyspark filter multiple conditions

Pyspark: Filter dataframe based on multiple conditions

stackoverflow.com › questions › 49301373 › pyspark-filter-dataframe-based-on-multiple-conditions

Your logic condition is wrong. IIUC, what you want is:

import pyspark.sql.functions as f

df.filter((f.col('d')<5))\
    .filter(
        ((f.col('col1') != f.col('col3')) | 
         (f.col('col2') != f.col('col4')) & (f.col('col1') == f.col('col3')))
    )\
    .show()

I broke the filter() step into 2 calls for readability, but you could equivalently do it in one line.

Output:

+----+----+----+----+---+
|col1|col2|col3|col4|  d|
+----+----+----+----+---+
|   A|  xx|   D|  vv|  4|
|   A|   x|   A|  xx|  3|
|   E| xxx|   B|  vv|  3|
|   F|xxxx|   F| vvv|  4|
|   G| xxx|   G|  xx|  4|
+----+----+----+----+---+

Answer from pault on Stack Overflow

Stack Overflow

stackoverflow.com › questions › 49301373 › pyspark-filter-dataframe-based-on-multiple-conditions

sql - Pyspark: Filter dataframe based on multiple conditions - Stack Overflow

Top answer

1 of 4

115

Your logic condition is wrong. IIUC, what you want is:

import pyspark.sql.functions as f

df.filter((f.col('d')<5))\
    .filter(
        ((f.col('col1') != f.col('col3')) | 
         (f.col('col2') != f.col('col4')) & (f.col('col1') == f.col('col3')))
    )\
    .show()

I broke the filter() step into 2 calls for readability, but you could equivalently do it in one line.

Output:

+----+----+----+----+---+
|col1|col2|col3|col4|  d|
+----+----+----+----+---+
|   A|  xx|   D|  vv|  4|
|   A|   x|   A|  xx|  3|
|   E| xxx|   B|  vv|  3|
|   F|xxxx|   F| vvv|  4|
|   G| xxx|   G|  xx|  4|
+----+----+----+----+---+

2 of 4

You can also write like below (without pyspark.sql.functions):

df.filter('d<5 and (col1 <> col3 or (col1 = col3 and col2 <> col4))').show()

Result:

+----+----+----+----+---+
|col1|col2|col3|col4|  d|
+----+----+----+----+---+
|   A|  xx|   D|  vv|  4|
|   A|   x|   A|  xx|  3|
|   E| xxx|   B|  vv|  3|
|   F|xxxx|   F| vvv|  4|
|   G| xxx|   G|  xx|  4|
+----+----+----+----+---+

python - Filter spark dataframe with multiple conditions on multiple columns in Pyspark

python - Sparksql filtering (selecting with where clause) with multiple conditions

Filter out rows in Spark dataframe based on condition

multiple conditions for filter in spark data frames

python - PySpark Dataframes: how to filter on multiple conditions with compact code?

More results from stackoverflow.com

Apache

spark.apache.org › docs › latest › api › python › reference › pyspark.sql › api › pyspark.sql.DataFrame.filter.html

pyspark.sql.DataFrame.filter — PySpark 4.2.0 documentation

Filter by multiple conditions using SQL expression.

Discussions

Filtering multiple conditions RDD

I'd make a single parsing function to do it in one pass: def get_month(text: str) -> Tuple[str, int]: for month in range(1,13): # Go through each month year_month = f'2020-{month:02}' # Make your string if year_month in text: # Check if it matches return (year_month, 1) # Couldn't find any month return ('0000-00', 1) a.map(get_month).filter(lambda x: x[0] != '0000-00') More on reddit.com

r/PySpark

March 31, 2021

Databricks Pyspark filter several columns with similar criteria

I am querying a table from the Databricks Catalog which I have to filter several columns with the same criteria. below is what I have created so far. I have 10 columns that I have filter with a set of criteria from (dx_list1) and another 10 that I have to filter with another set of criteria ... More on community.databricks.com

community.databricks.com

October 13, 2024

Panda - filter multiple columns by multiple values

You can use .isin(): RawData["Column3"].isin(["AAA", "BBB"]) More on reddit.com

r/learnpython

June 21, 2024

Filtering list of objects on multiple conditions

Hi all, interested in your approaches and suggestions to this problem. Let's say I have created a custom Player class that contains many attributes… More on reddit.com

r/learnpython

July 27, 2023

Videos