The problem is in you calling collect to store the result of a RDD here:

result = file.map(lambda x: x.split("\n")).collect()

This command will return to you a list, not an RDD. If you remove the collect() from this row like this:

result = file.map(lambda x: x.split("\n"))

this will work.

Collect() returns all the records of the Dataframe as a list of type Row. And you are calling 'SaveAsTextFile' on the result which is a list.

List doesnt have the 'saveAsTextFile' function, so it's throwing an error.

result = myresults.collect()
result.saveAsTextFile("test")

To save the contents of the Dataframe to file, you have 2 options:

1. Convert the DataFrame into RDD and call 'saveAsTextFile' function on it.

   myresults.rdd.saveAsTextFile(OUTPUT_PATH)

2. Using DataframeWriter. In this case, DataFrame must have only one column that is of string type. Each row becomes a new line in the output file.

   myresults.write.format("text").save(OUTPUT_PATH)

As you have more than 1 column in Dataframe, proceed with Option:1.

Also by default, spark will create 200 Partitions for shuffle. so, 200 files will be created in the output path. If you less data, configure the below parameter according to your data size.

spark.conf.set("spark.sql.shuffle.partitions", 5) # 5 files will be written to output folder.

map(filterOut2, data) works:

>>> data = [[1,2,3,5],[1,2,5,2],[3,5,2,8],[6,3,1,2],[5,3,2,5],[4,1,2,5] ]
... def filterOut2(line):
...     return [x for x in line if x != 2]
... list(map(filterOut2, data))
...
[[1, 3, 5], [1, 5], [3, 5, 8], [6, 3, 1], [5, 3, 5], [4, 1, 5]]

map() takes exactly 1 argument (2 given)

Looks like you redefined map. Try __builtin__.map(filterOut2, data).

Or, use a list comprehension:

>>> [filterOut2(line) for line in data]
[[1, 3, 5], [1, 5], [3, 5, 8], [6, 3, 1], [5, 3, 5], [4, 1, 5]]

Most probably your DataFrame is the Pandas DataFrame object, not Spark DataFrame object.

try:

Copyspark.createDataFrame(df).write.saveAsTable("dashboardco.AccountList")