You can try getItem(0):

df \
    .withColumn("CurrencyCode", df["CurrencyCode"].getItem(0).cast("string")) \
    .withColumn("TicketAmount", df["TicketAmount"].getItem(0).cast("string")) 

The final cast to string is optional.

You don't need Pandas for this. Spark has its own regex replace function.

This will replace \n in every row with an empty string.

By default, spark.read.text will read each line of the file into one dataframe row, so you cannot have a multi-line string value, anyway...

from pyspark.sql.functions import col, regexp_replace

df = spark.read.text("hdfs://test.txt")
df = df.select(regexp_replace(col('value'), '\n', ''))
df.show()

To get the dataframe into a joined string, collect the dataframe. But this should be avoided for large datasets.

s = '\n'.join(d['value'] for d in df.collect())

While you can use a UserDefinedFunction it is very inefficient. Instead it is better to use concat_ws function:

from pyspark.sql.functions import concat_ws

df.withColumn("test_123", concat_ws(",", "test_123")).show()

+----+----------------+
|uuid|        test_123|
+----+----------------+
|   1|test,test2,test3|
|   2|test4,test,test6|
|   3|test6,test9,t55o|
+----+----------------+

Just:

Copyfrom pyspark.sql.functions import col

table = spark.sql("table")

table.select([col(c).cast("string") for c in table.columns])

PySpark Row is just a tuple and can be used as such. All you need here is a simple map (or flatMap if you want to flatten the rows as well) with list:

data.map(list)

or if you expect different types:

data.map(lambda row: [str(c) for c in row])

You simply need to convert the list of string in the correct format like this:

# convert the list of string into proper format
>>> l = ' '.join(ListofString)
>>> l = l.replace(',',' ')
>>> l = [x.strip().split(' ') for x in l.split('\n')]

>>> print(l)

>>> [['Column1', 'Column2', 'Column3'], ['Col1Value1', 'Col2Value1', 'Col3Value1'], ['Col1Value2', 'Col2Value2', 'Col3Value2']]

>>> df = spark.createDataFrame(l[1:],l[0])

>>> df.show()

+----------+----------+----------+
|   Column1|   Column2|   Column3|
+----------+----------+----------+
|Col1Value1|Col2Value1|Col3Value1|
|Col1Value2|Col2Value2|Col3Value2|
+----------+----------+----------+