I was able to read ISO-8859-1 using spark but when I store the same data to S3/hdfs back and read it, the format is converting to UTF-8.

ex: é to é

val df = spark.read.format("csv").option("delimiter", ",").option("ESCAPE quote", '"'). option("header",true).option("encoding", "ISO-8859-1").load("s3://bucket/folder")

The difference in behavior you're observing is likely due to how the Spark DataFrameReader handles text files versus CSV files, particularly in how the encoding is applied during the read process.

I would approach this as following. First importing the required libraries:

import spark.implicits._
import org.apache.spark.sql.functions._

Then reading the text as a binary file that then we can apply the right encoding

val text = spark.read
  .format("binaryFile")
  .load("ISO_8859_1.txt")
  .select(col("content"))
  .as[Array[Byte]]
  .flatMap(bytes => new String(bytes, "ISO-8859-1")
  .split("\n")) // Split lines here
  .toDF("value")

text.show()

However while read properly,right now the data is still in one column

+-----------+
|      value|
+-----------+
|José, André|
+-----------+

So in order to create the dataframe as you want it you can do the following:

val csvData = text
  .withColumn("_tmp", split(col("value"), ","))
  .select(
    trim(col("_tmp").getItem(0)).as("_c0"),
    trim(col("_tmp").getItem(1)).as("_c1")
  )

csvData.show()

result:

+----+-----+
| _c0|  _c1|
+----+-----+
|José|André|
+----+-----+

Note: you will probably need to make this whole thing into a function or something and change it to taste. But I think its a good start.