This worked for me:

import os
import sys

spark_path = "D:\spark"

os.environ['SPARK_HOME'] = spark_path
os.environ['HADOOP_HOME'] = spark_path

sys.path.append(spark_path + "/bin")
sys.path.append(spark_path + "/python")
sys.path.append(spark_path + "/python/pyspark/")
sys.path.append(spark_path + "/python/lib")
sys.path.append(spark_path + "/python/lib/pyspark.zip")
sys.path.append(spark_path + "/python/lib/py4j-0.9-src.zip")

from pyspark import SparkContext
from pyspark import SparkConf

sc = SparkContext("local", "test")

To verify:

In [2]: sc
Out[2]: <pyspark.context.SparkContext at 0x707ccf8>

I'm assuming you already have spark and jupyter notebooks installed and they work flawlessly independent of each other.

If that is the case, then follow the steps below and you should be able to fire up a jupyter notebook with a (py)spark backend.

1. Go to your spark installation folder and there should be a bin directory there: /path/to/spark/bin

2. Create a file, let's call it start_pyspark.sh

3. Open start_pyspark.sh and write something like:

       #!/bin/bash
   
   export PYSPARK_PYTHON=/path/to/anaconda3/bin/python
   export PYSPARK_DRIVER_PYTHON=/path/to/anaconda3/bin/jupyter
   export PYSPARK_DRIVER_PYTHON_OPTS="notebook --NotebookApp.open_browser=False --NotebookApp.ip='*' --NotebookApp.port=8880"
   
   pyspark "$@"
   
   
   

Replace the /path/to ... with the path where you have installed your python and jupyter binaries respectively.

4. Most probably this step is already done, but just in case
   Modify your ~/.bashrc file by adding the following lines

       # Spark
       export PATH="/path/to/spark/bin:/path/to/spark/sbin:$PATH"
       export SPARK_HOME="/path/to/spark"
       export SPARK_CONF_DIR="/path/to/spark/conf"
   

Run source ~/.bashrc and you are set.

Go ahead and try start_pyspark.sh.
You could also give arguments to the script, something like start_pyspark.sh --packages dibbhatt:kafka-spark-consumer:1.0.14.

Hope it works out for you.