SparkContext doesn't have, SQLContext has:
from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
sqlContext.createDataFrame(pandas_df)
SparkSession is not a replacement for a SparkContext but an equivalent of the SQLContext. Just use it use the same way as you used to use SQLContext:
spark.createDataFrame(...)
and if you ever have to access SparkContext use sparkContext attribute:
spark.sparkContext
so if you need SQLContext for backwards compatibility you can:
SQLContext(sparkContext=spark.sparkContext, sparkSession=spark)
Whenever we are trying to create a DF from a backward-compatible object like RDD or a data frame created by spark session, you need to make your SQL context-aware about your session and context.
Like Ex:
If I create a RDD:
ss=SparkSession.builder.appName("vivek").master('local').config("k1","vi").getOrCreate()
rdd=ss.sparkContext.parallelize([('Alex',21),('Bob',44)])
But if we wish to create a df from this RDD, we need to
sq=SQLContext(sparkContext=ss.sparkContext, sparkSession=ss)
then only we can use SQLContext with RDD/DF created by pandas.
schema = StructType([
StructField("name", StringType(), True),
StructField("age", IntegerType(), True)])
df=sq.createDataFrame(rdd,schema)
df.collect()
This should work (except in the code you have a missing ')' in the end of sc creation which I imagine is a type). You can try creating sc as follows:
conf = SparkConf().setAppName("app1").setMaster("local")
sc = SparkContext(conf=conf)
BTW sc.stop means you already have a spark context which is true if you used pyspark but not if you use spark-submit. It is better to use SparkContext.getOrCreate which works in both cases.
Here is a minimal example that worked for me. I am not sure, why you imported pandas in the first place, if you are not using it afterwards. I guess your intention was to create a DataFrame from a pandas object. Therefore here is an example to generate a spark-DataFrame from a pandas-Dataframe.
import pandas as pd
from pyspark import SQLContext
df = pd.DataFrame({'x': [1, 2, 3]})
sc = SparkContext.getOrCreate()
sqlContext = SQLContext(sc)
sqlContext.createDataFrame(df)
I am running spark in a jupyter notebook too.