Pyspark will not decode correctly if the hex vales are preceded by double backslashes (ex: \\xBA instead of \xBA).

Using "take(3)" instead of "show()" showed that in fact there was a second backslash:

[Row(item_name='Jogador n\\xBA 10'),
 Row(item_name='Camisa N\\xB0 9'),
 Row(item_name='Uniforme M\\xE9dio')]

To solve this I created a UDF to decode using "unicode-escape" method:

import pyspark.sql.functions as F
import pyspark.sql.types as T
my_udf = F.udf(lambda x: x.encode().decode('unicode-escape'),T.StringType())
df.withColumn('test', my_udf('item_name')).show()
+------------------+---------------+
|         item_name|           test|
+------------------+---------------+
|  Jogador n\xBA 10|  Jogador nº 10|
|    Camisa N\xB0 9|    Camisa N° 9|
| Uniforme M\xE9dio| Uniforme Médio|
+------------------+---------------+

It is quite possible here that your encoding gets bypassed and not working. Try using any of the aliases as "utf8, latin-1, latin1, iso-8859-1, iso8859-1" instead of "utf-8".

See the related docs as : "https://docs.python.org/3/library/codecs.html#standard-encodings"

For example replace: option("encoding", "utf-8") with ("encoding","iso-8859-1")

So this workaround helped to solve this, By changing the default encoding for the session

import sys
reload(sys)
sys.setdefaultencoding('UTF-8')

and then

df_pandas_df = df_pandas_df.astype(str)

converts whole dataframe as string df.

https://issues.apache.org/jira/browse/SPARK-11772 talks about this issue and gives a solution that runs:

export PYTHONIOENCODING=utf8

before running pyspark. I wonder why above works, because sys.getdefaultencoding() returned utf-8 for me even without it.

How to set sys.stdout encoding in Python 3? also talks about this and gives the following solution for Python 3:

import sys
sys.stdout = open(sys.stdout.fileno(), mode='w', encoding='utf8', buffering=1)

Try encoding it in UTF-8 and make sure you read your text file with the appropriate encoding:

df.coalesce(1).write.option("header", "true").option("encoding", "UTF-8").csv(r'path\....\file.csv')

You have a binary type column, which is like a bytearray in python. You just need to cast to string:

df = df.withColumn("C2", df["C2"].cast("string"))
df.show()
#+---+---+---+
#| C1| C2| C3|
#+---+---+---+
#|  1| --|  0|
#|  2|201|  0|
#|  3|Com|  0|
#+---+---+---+

Likewise in python:

str(bytearray([0x20, 0x2D, 0x2D]))
#' --'