Here is one way (basically melt the DF, then pivot)

# First combine price and units into a map column
mydf = mydf.withColumn("price_units", F.create_map(F.lit("price"), "price", F.lit("units"), "units"))

# Now explode to get a melted dataframe
mydf = mydf.select("id", "day", F.explode("price_units").alias("name", "value"))

+---+---+-----+-----+
| id|day| name|value|
+---+---+-----+-----+
|100|  1|price|   23|
|100|  1|units|   10|
|100|  2|price|   45|
|100|  2|units|   11|
|100|  3|price|   67|
etc

# Then pivot
mydf.groupby("id", "name").pivot("day").agg(F.mean("value")).show()

+---+-----+----+----+----+----+
| id| name|   1|   2|   3|   4|
+---+-----+----+----+----+----+
|100|price|23.0|45.0|67.0|78.0|
|101|price|23.0|45.0|67.0|78.0|
|102|units|10.0|11.0|16.0|18.0|
|100|units|10.0|11.0|12.0|13.0|
|101|units|10.0|13.0|14.0|15.0|
|102|price|23.0|45.0|67.0|78.0|
+---+-----+----+----+----+----+

Well, I managed to make it work, but ... I don't understand why it fixes the issue. I encourage Spark Dev team to have a look at why this works better than the naive approach.

I had a group of several aggregation lists of lists, rather than sending all of them to pivot I computed them one by one and then joined the result. This ran much faster, about 100 seconds instead of 500: Each item in agg_list is a list of functions on a family type (e.g. num/str etc) with aliases e.g. f.sum(f.col(col_name)).alias(col_name)

df_groupby = df.groupby(groupby_col)
df_pivot = df_groupby.pivot(pivot_col, pivot_distinct_values)
pivot_list = [df_pivot.agg(*a) for a in agg_list] 
pivot_join = join_dataframes(pivot_list, on=self.user_id_col)

where join_dataframes is a small utility function:

def join_dataframes(df_list: List[pyspark.sql.DataFrame], *args, **kwargs) -> pyspark.sql.DataFrame:
    """joins multiple :class:`DataFrame` with the same `on` and `how`

    pass on, how etc like in join(other, on=None, how=None) method of pyspark.sql.dataframe.DataFrame instance

    :param df_list: a list of dataframes
    :type df_list: List[pyspark.sql.DataFrame]
    :return: the joined pyspark dataframe
    :rtype: pyspark.sql.DataFrame
    """

    df_joined = df_list[0]
    for df in df_list[1:]:
        df_joined.join(df, *args, **kwargs)
    return df_joined