Had the same error. You have to run appbuilder.init_app() for your app:
..
..
class RubyAPP(object):
def __init__(self, config_name):
# App manager
app.config.from_pyfile('config.py')
db.init_app(app)
with app.app_context():
appbuilder.init_app(app, db.session)
..
..
Answer from frans on Stack Overflowpython - pyspark error: AttributeError: 'SparkSession' object has no attribute 'parallelize' - Stack Overflow
AttributeError: 'builtin_function_or_method' object has no attribute
As I'm not getting much help with my original question I did some research and rewrote the pyspark.sql query as follows:
#%%import findsparkfindspark.init('/home/packt/spark-2.1.0-bin-hadoop2.7')from pyspark.sql import SparkSessionspark = SparkSession.builder.appName('ops').getOrCreate()df = spark.read.csv('/home/packt/Downloads/Spark_DataFrames/Person_Person.csv',inferSchema=True,header=True)df.createOrReplaceTempView('Person_Person')myresults = spark.sql("""SELECTPersonType,COUNT(PersonType) AS \Person Count`FROM Person_PersonGROUP BY PersonType""")myresults.collect()result = myresults.collect()resultresult.saveAsTextFile("test")`
However, I'm now getting the following error message:
AttributeError Traceback (most recent call last) <ipython-input-9-9e137ed161cc> in <module>() ----> 1 result.saveAsTextFile("test") AttributeError: 'list' object has no attribute 'saveAsTextFile'
Hopefully, this will get more responses to help me fix this issue
Thanks
More on reddit.comAttributeError: 'tensorrt.tensorrt.Builder' object has no attribute 'fp16_mode'
Trouble with spark code in Notebook, 'str' object has no attribute 'option
SparkSession is not a replacement for a SparkContext but an equivalent of the SQLContext. Just use it use the same way as you used to use SQLContext:
spark.createDataFrame(...)
and if you ever have to access SparkContext use sparkContext attribute:
spark.sparkContext
so if you need SQLContext for backwards compatibility you can:
SQLContext(sparkContext=spark.sparkContext, sparkSession=spark)
Whenever we are trying to create a DF from a backward-compatible object like RDD or a data frame created by spark session, you need to make your SQL context-aware about your session and context.
Like Ex:
If I create a RDD:
ss=SparkSession.builder.appName("vivek").master('local').config("k1","vi").getOrCreate()
rdd=ss.sparkContext.parallelize([('Alex',21),('Bob',44)])
But if we wish to create a df from this RDD, we need to
sq=SQLContext(sparkContext=ss.sparkContext, sparkSession=ss)
then only we can use SQLContext with RDD/DF created by pandas.
schema = StructType([
StructField("name", StringType(), True),
StructField("age", IntegerType(), True)])
df=sq.createDataFrame(rdd,schema)
df.collect()
Hello community,
I am trying to collect and send the results from a pyspark query to a textfile.
However, I keep on getting the error:
AttributeError: 'builtin_function_or_method' object has no attribute example8
I'm extremely new to pyspark.sql. The code is as follows:
#%%
import sys
from operator import add
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('aggs').getOrCreate()
df = spark.read.csv('/home/packt/Downloads/Spark_DataFrames/sales_info.csv',inferSchema=True,header=True)
example8 = spark.sql("""SELECT
*
FROM sales_info
ORDER BY Sales DESC""")
print.example8.collect()
example8.saveAsTextFile("/home/packt/test.txt")
read_rdd = sc.textFile("/home/packt/test.txt")
read_rdd.collect()
main()
The full error message is as follows:
Append ResultsClear Results
--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-42-714a9bbd2b92> in <module>() 74 FROM sales_info 75 ORDER BY Sales DESC""") ---> 76 print.example8.collect() 77 78 example8.saveAsTextFile("/home/packt/test.txt") AttributeError: 'builtin_function_or_method' object has no attribute 'example8'Any help figuring out the error will be greatly appreciated.
Thanks
Your indentation is goofed, and you've mixed tabs and spaces. Run the script with python -tt to verify.
If you’re using python 3+ this may also occur if you’re using private variables that start with double underscore, e.g., self.__yourvariable. Just something to take note of for some of you who may run into this issue.