I am happy now because I have been having exactly the same issue with my pyspark and I found "the solution". In my case, I am running on Windows 10. After many searches via Google, I found the correct way of setting the required environment variables:
PYTHONPATH=$SPARK_HOME$\python;$SPARK_HOME$\python\lib\py4j-<version>-src.zip
The version of Py4J source package changes between the Spark versions, thus, check what you have in your Spark and change the placeholder accordingly.
For a complete reference to the process look at this site: how to install spark locally
I am happy now because I have been having exactly the same issue with my pyspark and I found "the solution". In my case, I am running on Windows 10. After many searches via Google, I found the correct way of setting the required environment variables:
PYTHONPATH=$SPARK_HOME$\python;$SPARK_HOME$\python\lib\py4j-<version>-src.zip
The version of Py4J source package changes between the Spark versions, thus, check what you have in your Spark and change the placeholder accordingly.
For a complete reference to the process look at this site: how to install spark locally
For me
import findspark
findspark.init()
import pyspark
solved the problem
Hey everyone!
I recently started working with Apache Spark, and its PySpark implementation in a professional environment, thus I am by no means an expert, and I am facing an error with Py4J.
In more details, I have installed Apache Spark, and already set up the SPARK_HOME, HADOOP_HOME, JAVA_HOME environment variables. As I want to run PySpark without using pip install pyspark, I have set up a PYTHONPATH environment variable, with values pointing to the python folder of Apache Spark and inside the py4j.zip.
My issue is that when I create a dataframe from scratch and use the command df.show() I get the Error
*"*Py4JJavaError: An error occurred while calling o143.showString. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 4.0 failed 1 times, most recent failure: Lost task 0.0 in stage 4.0 (TID 4) (xxx-yyyy.mshome.net executor driver): org.apache.spark.SparkException: Python worker failed to connect back".
However, the command works as it should when the dataframe is created, for example, by reading a csv file. Other commands that I have also tried, works as they should.
The version of the programs that I use are:
Python 3.11.9 (always using venv, so Python is not in path)
Java 11
Apache Spark 3.5.1 (and Hadoop 3.3.6 for the win.utls file and hadoop.dll)
Visual Studio Code
Windows 11
I have tried other version of Python (3.11.8, 3.12.4) and Apache Spark (3.5.2), with the same response
Any help would be greatly appreciated!
The following two pictures just show an example of the issue that I am facing.
----------- UPDATED SOLUTION -----------
In the end, also thanks to the suggestions in the comments, I figured out a way to make PySpark work with the following implementation. After running this code in a cell, PySpark is recognized as it should and the code runs without issues even for the manually created dataframe, Hopefully, it can also be helpful to others!
# Import the necessary libraries
import os, sys
# Add the necessary environment variables
os.environ["PYSPARK_PYTHON"] = sys.executable
os.environ["spark_python"] = os.getenv('SPARK_HOME') + "\\python"
os.environ["py4j"] = os.getenv('SPARK_HOME') + "\\python\lib\py4j-0.10.9.7-src.zip"
# Retrieve the values from the environment variables
spark_python_path = os.environ["spark_python"]
py4j_zip_path = os.environ["py4j"]
# Add the paths to sys.path
for path in [spark_python_path, py4j_zip_path]:
if path not in sys.path:
sys.path.append(path)
# Verify that the paths have been added to sys.path
print("sys.path:", sys.path)