You may try the following which uses explode to split the list of Employee Ids into different rows before joining them and using collect_list to aggregate the entries into a list.
Using spark sql:
NB. Ensure Employees is available as a table/view eg EmployeeData.createOrReplaceTempView("Employees")
WITH dept_employees AS (
SELECT
DeptId,
DeptName,
explode(EmployeeIDs)
FROM
vw_TestView
)
SELECT
d.DeptId,
d.DeptName,
collect_list(e.EmpID) as EmployeeIDs,
collect_list(e.EmpName) as EmployeeNames
FROM
dept_employees d
INNER JOIN
Employees e ON d.col=e.EmpID
GROUP BY
d.Deptid,
d.DeptName
or using pyspark api:
from pyspark.sql import functions as F
output_df = (
departMentData.select(
F.col("DeptId"),
F.col("DeptName"),
F.explode("EmployeeIDs")
)
.alias("d")
.join(
EmployeeData.alias("e"),
F.col("d.col")==F.col("e.EmpID"),
"inner"
)
.groupBy("d.DeptId","d.DeptName")
.agg(
F.collect_list("e.EmpID").alias("EmployeeIDs"),
F.collect_list("e.EmpName").alias("EmployeeNames")
)
)
Let me know if this works for you.
Answer from ggordon on Stack OverflowDatabricks
docs.databricks.com › sql language reference › functions › built-in functions › alphabetical list of built-in functions › array_join function
array_join function | Databricks on AWS
September 11, 2026 - > SELECT array_join(array('hello', 'world'), ','); hello,world > SELECT array_join(array('hello', NULL ,'world'), ','); hello,world > SELECT array_join(array('hello', NULL ,'world'), ',', '*'); hello,*,world
Databricks
api-docs.databricks.com › python › pyspark › latest › pyspark.sql › api › pyspark.sql.functions.array_join.html
pyspark.sql.functions.array_join — PySpark master documentation
Concatenates the elements of column using the delimiter. Null values are replaced with null_replacement if set, otherwise they are ignored · pyspark.sql.functions.slice
Microsoft Learn
learn.microsoft.com › en-us › azure › databricks › pyspark › reference › functions › array_join
array_join - Azure Databricks | Microsoft Learn
June 4, 2026 - array\_join function in PySpark: Returns a string column by concatenating the elements of the input array column using the delimiter.
Databricks
docs.databricks.com › pyspark reference › functions › array_join
array_join | Databricks on AWS
June 4, 2026 - array\_join function in PySpark: Returns a string column by concatenating the elements of the input array column using the delimiter.
Stack Overflow
stackoverflow.com › questions › 77586299 › databricks-sql-array-join
apache spark sql - Databricks SQL array join - Stack Overflow
I have created query in databricks sql thats grouping text values by certain key. The problem is that in some cases different date in grouping key value creates 2 separate rows ... Final AS ( SELECT ,1.PRIMARY_KEY ,1.DUMMYCOLUMN ,1.DATE_START ,1.DATE_END ,ARRAY_JOIN(COLLECT_SET(Sales), ', ') AS TEST FROM Contract GROUP BY PRIMARY_KEY, DATE_START, DATE_END) SELECT * FROM FINAL
Databricks
databricks-prod-cloudfront.cloud.databricks.com › public › 4027ec902e239c93eaaa8714f173bcfc › 142158605138935 › 3773509768457258 › 7497868276316206 › latest.html
Examples of New Built-in Functions and Higher-order Functions - Databricks
%md #### array_join(array<String>, String[, String]): String Concatenates the elements of the given array using the delimiter and an optional string to replace nulls.
Databricks Documentation
docs.databricks.com › pyspark reference › functions › array_join
array_join | Databricks on Google Cloud
array\_join function in PySpark: Returns a string column by concatenating the elements of the input array column using the delimiter.
Databricks
docs.databricks.com › aws › en › notebooks › source › apache-spark-functions.html
apache-spark-2.4-functions - Databricks
%md #### array_join(array<String>, String[, String]): String Concatenates the elements of the given array using the delimiter and an optional string to replace nulls.
Databricks
docs.databricks.com › sql language reference › functions › built-in functions › alphabetical list of built-in functions › array_union function
array_union function | Databricks on AWS
Returns an array of the elements in the union of array1 and array2 without duplicates.
Azure Docs
docs.azure.cn › en-us › databricks › sql › language-manual › functions › array_join
array_join function - Azure Databricks | Azure Docs
April 10, 2024 - Concatenates the elements of array.
Databricks Community
community.databricks.com › databricks community › data engineering › is it possible to control the ordering of the array values created by array_agg()?
Is it possible to control the ordering of the array values created by array_agg()?
April 10, 2024 - with tmp as ( select * from (values (1, 1, 'cherry'), (1, 3, 'strawberries'), (1, 2, 'acerola'), (1, 5, 'banan'), (1, 4, 'lemon') ) as tab(custid, hist, alf) order by custid asc, hist asc ) select custid, array_join(array_agg(alf), ' -> ') as move from tmp group by custid order by custid asc;
Databricks
databricks-prod-cloudfront.cloud.databricks.com › public › 4027ec902e239c93eaaa8714f173bcfc › 6907863442366616 › 2357259023612622 › 1911168149879216 › latest.html
8. Joins - Databricks
import org.apache.spark.sql.functions.expr person.withColumnRenamed("id", "personId") .join(sparkStatus, expr("array_contains(spark_status, id)")).show()
Microsoft Learn
learn.microsoft.com › en-us › azure › databricks › sql › language-manual › functions › concat
concat function - Azure Databricks - Databricks SQL | Microsoft Learn
June 24, 2026 - When concatenating arrays, Azure Databricks raises COLLECTION_SIZE_LIMIT_EXCEEDED if the result exceeds the array size limit. COLLECTION_SIZE_LIMIT_EXCEEDED · > SELECT concat('Spark', 'SQL'); SparkSQL > SELECT concat(array(1, 2, 3), array(4, 5), array(6)); [1,2,3,4,5,6] || (pipe pipe sign) operator · array_join function ·
Databricks Community
community.databricks.com › databricks community › data engineering › use array in where in clause
Solved: Use Array in WHERE IN clause - Databricks Community - 129326
August 25, 2025 - In your SQL statement, where IDs in (ID) is comparing a string column (IDs) with an array variable (ARRAY<STRING>). Databricks SQL does not allow IN to directly take an array — IN expects a list of scalar values, not an array object.
Databricks Community
community.databricks.com › t5 › data-engineering › error-while-using-array-contains-function-in-left-join-condition › td-p › 7441
Error while using Array_contains function in left join condition
March 27, 2023 - 'Item_id' is column in array format like ["ba1b-5fbe1547ddd5", "88f9-ac3b93334f69", "8bba-4075a47eb814"] in table1 and table2 has column Id with single value like ba1b-5fbe1547ddd5. While join two table select table1.*, table2.* from table1 left join table2 on array_contains(table2.id, table1.ite...