Looks like you might have inconsistencies with your Spark versions installed.

You have your Pyspark code that tries to call the legacyInferArrayTypeFromFirstElement method of the underlying SQLConf object, which has only been introduced since since v3.4.0.

But since your error is

py4j.Py4JException: Method legacyInferArrayTypeFromFirstElement([]) does not exist

I would think that your underlying Spark installation is not on version 3.4.0. This is of course dependent on how you have Spark installed so it's hard to say exactly. Try to verify which version your Pyspark is using (should be 3.4.0) and which version of Spark the executors start up with.

For posterity, you can also get a similar error if you pass non-Column objects in the expression. For example:

column = 'A'
df.select(df[column] == 0)  # this is fine

column = ['A']              # whoops, df[['A']] results in a DataFrame, not a Column
df.select(df[column] == 0)  # py4j.Py4JException: Method col([class java.lang.Boolean]) does not exist