You can use df.rdd.map(), as DataFrame does not have map or flatMap, but be aware of the implications of using df.rdd:

Converting to RDD breaks Dataframe lineage, there is no predicate pushdown, no column prunning, no SQL plan and less efficient PySpark transformations.

What should you do instead?

Keep in mind that the high-level DataFrame API is equipped with many alternatives. First, you can use select or selectExpr.

Another example is using explode instead of flatMap(which existed in RDD):

df.select($"name",explode($"knownLanguages"))
    .show(false)

Result:

+-------+------+
|name   |col   |
+-------+------+
|James  |Java  |
|James  |Scala |
|Michael|Spark |
|Michael|Java  |
|Michael|null  |
|Robert |CSharp|
|Robert |      |
+-------+------+

You can also use withColumn or UDF, depending on the use-case, or another option in the DataFrame API.

Main problem is after selecting old column get DataFrame instead Series, so map implemented yet to Series failed.

Here should be duplicated column old, so if select one column it return all columns old in DataFrame:

df = pd.DataFrame([[1,3,8],[4,5,3]], columns=['old','old','col'])
print (df)
   old  old  col
0    1    3    8
1    4    5    3

print(df['old'])
   old  old
0    1    3
1    4    5

#dont use dict like variable, because python reserved word
df['new'] = df['old'].map(d)
print (df)

AttributeError: 'DataFrame' object has no attribute 'map'

Possible solution for deduplicated this columns:

s = df.columns.to_series()
new = s.groupby(s).cumcount().astype(str).radd('_').replace('_0','')
df.columns += new
print (df)
   old  old_1  col
0    1      3    8
1    4      5    3

Another problem should be MultiIndex in column, test it by:

mux = pd.MultiIndex.from_arrays([['old','old','col'],['a','b','c']])
df = pd.DataFrame([[1,3,8],[4,5,3]], columns=mux)
print (df)
  old    col
    a  b   c
0   1  3   8
1   4  5   3

print (df.columns)
MultiIndex(levels=[['col', 'old'], ['a', 'b', 'c']],
           codes=[[1, 1, 0], [0, 1, 2]])

And solution is flatten MultiIndex:

#python 3.6+
df.columns = [f'{a}_{b}' for a, b in df.columns]
#puthon bellow
#df.columns = ['{}_{}'.format(a,b) for a, b in df.columns]
print (df)
   old_a  old_b  col_c
0      1      3      8
1      4      5      3

Another solution is map by MultiIndex with tuple and assign to new tuple:

df[('new', 'd')] = df[('old', 'a')].map(d)
print (df)
  old    col new
    a  b   c   d
0   1  3   8   A
1   4  5   3   D

print (df.columns)
MultiIndex(levels=[['col', 'old', 'new'], ['a', 'b', 'c', 'd']],
           codes=[[1, 1, 0, 2], [0, 1, 2, 3]])