Scala Spark functions library do not have these function but spark sql librry do have these functions. This is why you are not able to use as spark function API.

https://spark.apache.org/docs/2.0.2/api/java/org/apache/spark/sql/functions.html

isNull function exists that can be combined with when/then clause to set values.

I hope it helps.

Answer from Ramdev Sharma on Stack Overflow
🌐
Apache
spark.apache.org › docs › latest › api › python › reference › pyspark.sql › api › pyspark.sql.functions.nullif.html
pyspark.sql.functions.nullif - Apache Spark
pyspark.sql.functions.nullif(col1, col2)[source]# Returns null if col1 equals to col2, or col1 otherwise. New in version 3.5.0. Parameters · col1Column or str · col2Column or str · Examples · >>> df = spark.createDataFrame([(None, None,), (1, 9,)], ["a", "b"]) >>> df.select(nullif(df.a, ...
🌐
Databricks Documentation
docs.databricks.com › reference › sql language reference › functions › built-in functions › alphabetical list of built-in functions › nullif function
nullif function | Databricks on Google Cloud
Applies to: Databricks SQL Databricks Runtime · Returns NULL if expr1 equals expr2, or expr1 otherwise. nullif(expr1, expr2) expr1: An expression of any type. expr2: An expression of the same type as expr.
🌐
Spark Code Hub
sparkcodehub.com › spark › dataframe › coalesce-nullif
Handling Nulls with Coalesce and NullIf in Spark DataFrames: A Complete Guide in Scala
NULLIF expressions handle nulls and invalid statuses, integrating with SQL workflows (Spark DataFrame SelectExpr Guide).
🌐
Databricks Documentation
docs.databricks.com › reference › sql language reference › functions › built-in functions › alphabetical list of built-in functions › nullif function
nullif function | Databricks on AWS
Applies to: Databricks SQL Databricks Runtime · Returns NULL if expr1 equals expr2, or expr1 otherwise. nullif(expr1, expr2) expr1: An expression of any type. expr2: An expression of the same type as expr.
🌐
GitHub
github.com › apache › spark › blob › master › sql › catalyst › src › main › scala › org › apache › spark › sql › catalyst › expressions › nullExpressions.scala
spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala at master · apache/spark
case class NullIf(left: Expression, right: Expression, replacement: Expression) extends RuntimeReplaceable with InheritAnalysisRules { · def this(left: Expression, right: Expression) = { this(left, right, if (!SQLConf.get.getConf(SQLConf.ALWAYS_INLINE_COMMON_EXPR)) { With(left) { case Seq(ref) => If(EqualTo(ref, right), Literal.create(null, left.dataType), ref) } } else { If(EqualTo(left, right), Literal.create(null, left.dataType), left) } ) } ·
Author   apache
Find elsewhere
🌐
Apache
spark.apache.org › docs › latest › api › python › reference › pyspark.sql › api › pyspark.sql.functions.ifnull.html
pyspark.sql.functions.ifnull - Apache Spark
>>> import pyspark.sql.functions as sf >>> df = spark.createDataFrame([(None,), (1,)], ["e"]) >>> df.select(sf.ifnull(df.e, sf.lit(8))).show() +------------+ |ifnull(e, 8)| +------------+ | 8| | 1| +------------+ Show Source
🌐
Apache Spark
spark.apache.org › docs › latest › api › sql › index.html
Spark SQL, Built-in Functions
The function returns NULL if the key is not contained in the map. ... elt(n, input1, input2, ...) - Returns the n-th input, e.g., returns input2 when n is 2. The function returns NULL if the index exceeds the length of the array and spark.sql.ansi.enabled is set to false.
🌐
Databricks
docs.databricks.com › reference › sql language reference › functions › built-in functions › alphabetical list of built-in functions › nullifzero function
nullifzero function | Databricks on AWS
nullifzero(expr) expr: A numeric expression or NULL. The result type is the same as the type of expr. SQL · > SELECT nullifzero(0); NULL > SELECT nullifzero(NULL); NULL > SELECT nullifzero(5); 5 · if function · zeroifnull function · Syntax ...
🌐
Kontext
kontext.tech › home › code snippets & tips › spark sql - isnull and isnotnull functions
Spark SQL - isnull and isnotnull Functions - Kontext
July 9, 2022 - spark-sql> SELECT t.key, t.value, case when t.value is null then true else false end as is_null > FROM VALUES > ('a',1), > ('b',NULL) > AS t(key, value); a 1 false b NULL true
🌐
MungingData
mungingdata.com › apache-spark › dealing-with-null
Dealing with null in Spark - MungingData
Remember that DataFrames are akin to SQL databases and should generally follow SQL best practices. Scala best practices are completely different. The Databricks Scala style guide does not agree that null should always be banned from Scala code and says: "For performance sensitive code, prefer null over Option, in order to avoid virtual method calls and boxing." The Spark source code uses the Option keyword 821 times, but it also refers to null directly in code like if (ids != null).
🌐
Medium
medium.com › @uzzaman.ahmed › pyspark-normal-and-misc-functions-a-comprehensive-guide-fb6e2c61fb77
PySpark Normal and Misc Functions: A Comprehensive Guide | by Ahmed Uz Zaman | Medium
March 23, 2023 - from pyspark.sql.functions import nullif from pyspark.sql import SparkSession spark = SparkSession.builder.appName("NullIf Example").getOrCreate() data = [("apple", 5), ("orange", 0), ("banana", 10)] df = spark.createDataFrame(data, ["fruit", "count"]) df = df.withColumn("count_null", nullif(df["count"], 0)) df.show() # Output +------+-----+----------+ | fruit|count|count_null| +------+-----+----------+ | apple| 5| 5| |orange| 0| null| |banana| 10| 10| +------+-----+----------+
🌐
AWS
docs.aws.amazon.com › aws clean rooms › sql reference › aws clean rooms spark sql › aws clean rooms spark sql functions › conditional expressions › nullif function
NULLIF function - AWS Clean Rooms
select nullif(listid,salesid), salesid from sales where salesid<10 order by 1, 2 desc; listid | salesid --------+--------- 4 | 2 5 | 4 5 | 3 6 | 5 10 | 9 10 | 8 10 | 7 10 | 6 | 1 (9 rows)
🌐
Microsoft Learn
learn.microsoft.com › en-us › azure › databricks › sql › language-manual › functions › nullif
nullif function - Azure Databricks - Databricks SQL | Microsoft Learn
Applies to: Databricks SQL Databricks Runtime · Returns NULL if expr1 equals expr2, or expr1 otherwise. nullif(expr1, expr2) expr1: An expression of any type. expr2: An expression of the same type as expr.