Use ~ as bitwise NOT:
df2.where(~F.col('Key').contains('sd')).show()
Answer from mck on Stack OverflowFiltering rows that does not contain a string
Is there a way to filter a field not containing something in a spark dataframe using scala? - Stack Overflow
PySpark: Filtering for a value not working?
Pyspark dataframe operator "IS NOT IN" - Stack Overflow
You can negate predicate using either not or ! so all what's left is to add another condition:
import org.apache.spark.sql.functions.not
df.where($"referrer".contains("www.mydomain.") &&
not($"referrer".contains("google")))
or separate filter:
df
.where($"referrer".contains("www.mydomain."))
.where(!$"referrer".contains("google"))
You may use a Regex. Here you can find a reference for the usage of regex in Scala. And here you can find some hints about how to create a proper regex for URLs.
Thus in your case you will have something like:
val regex = "PUT_YOUR_REGEX_HERE".r // something like (https?|ftp)://www.mydomain.com?(/[^\s]*)? should work
val filteredDf = unfilteredDf.filter(regex.findFirstIn(($"referrer")) match {
case Some => true
case None => false
} )
This solution requires a bit of work but is the safest one.
Hi all, thanks for taking the time try and help me.
So this seems pretty basic but I'm really struggling. I have a pyspark dataframe that has a column called something_id and the ids in the column are a mixture of capital letters, lowercase letters and numbers. For example CyVmB5kL. I can see when I do df.show() that the something_id as a value called CyVmB5kL, but when I try df.filter("something_id = 'CyVmB5kL'").show() I get an empty dataframe. There's no whitespace, I've literally copied and pasted different values from df.show(). I've created my own df from stratch and tried the exact same steps and it worked as expected. So I'm lost and stackoverflow.com, chat-gpt and google have all failed me. You're my only hope. Any ideas?
Thanks in advanced
data_scallion
In pyspark you can do it like this:
array = [1, 2, 3]
dataframe.filter(dataframe.column.isin(array) == False)
Or using the binary NOT operator:
dataframe.filter(~dataframe.column.isin(array))
Take the operator ~ which means contrary :
df_filtered = df.filter(~df["column_name"].isin([1, 2, 3]))