It has some problems with Pylance and Microsoft Language Server. They are not smart enough. Could you have a try on JediLSP Language Server?
"python.languageServer": "JediLSP",
Update:
You can try to take 'import pyspark.sql' or 'from pyspark.sql import SparkSession' instead of 'import pyspark'.
It was designed like this. You can refer to this page for the explanation of the developer on this design.
Answer from Steven-MSFT on Stack OverflowGitHub
github.com › DonJayamanne › pythonVSCode › wiki › PySpark
PySpark · DonJayamanne/pythonVSCode Wiki
November 28, 2016 - Create environment variable SPARK_HOME point to installation home and add installation bin directory to global path ... Ctrl+Shift+P to activate command "Python: Update Workspace PySpark Libraries", this enables intellisense...
Author DonJayamanne
Videos
02:47
How to Install PySpark in VS Code (Visual Studio Code) (2025) - ...
Setting up PySpark in Visual Studio Code (with test module)
02:53
How to Install PySpark in Visual Studio Code (Easy) - YouTube
09:27
How to Run PySpark in Visual Studio Code | Pyspark | Apache Spark ...
01:31
Enable autocomplete or intellisense for Python in Visual Studio ...
GitHub
github.com › microsoft › vscode › issues › 223326
Intellisense not working · Issue #223326 · microsoft/vscode
July 23, 2024 - Except when I type the pyspark code, the intellisense should give auto suggestions. But it isn't providing it. Not even auto complete too.
Author baarthe
Medium
medium.com › @goelanirudh26 › the-10-best-vs-code-extensions-for-python-and-pyspark-developers-f65e5e1ab80a
The 10 Best VS Code Extensions for Python and PySpark Developers | by Anirudh Goel | Medium
March 2, 2023 - The PySpark IntelliSense extension provides code completion and documentation for PySpark functions and classes, while the Python Test Explorer and Code Runner extensions help you run and debug your code.
Top answer 1 of 3
2
Autocomplete works! I'am using macOS, but steps should be the same on Windows. Here is what I've done to get it:
1) Install Visual Studio IntelliCode extension
2)Quit both SparkAr and VSCode.
3)Open SparkAr and from there open your script.
And you will get it!
2 of 3
0
It's an old question, but if someone gets here when searching for the same problem in a more recent version (v101 in my case) here is one solution I found. In this version, the content in the folder '/private/var/folders/.../skylight-typedefs' is not included for some reason (the path can be found in the tsconfig.json that is created when adding a script to a project), so the IDE can't find the type definitions. I installed an older version and manually copied the files and added them back after upgrading to 101, then it worked.
Visual Studio Marketplace
marketplace.visualstudio.com › items
Spark & Hive Tools - Visual Studio Marketplace
Extension for Visual Studio Code - Spark & Hive Tools - PySpark Interactive Query, PySpark Batch, Hive Interactive Query, Hive Batch
Visual Studio Code
code.visualstudio.com › docs › editing › intellisense
IntelliSense
November 3, 2021 - VS Code IntelliSense features are powered by a language service. A language service provides intelligent code completions based on language semantics and an analysis of your source code.
Microsoft Azure
azure.microsoft.com › blog home › developer tools › run your pyspark interactive query and batch job in visual studio code
Run your PySpark Interactive Query and batch job in Visual Studio Code | Microsoft Azure Blog
June 26, 2025 - Built-in Python language service, such as IntelliSense auto-suggest, autocomplete, and error marker, among others. Preview and export your PySpark interactive query results to CSV, JSON, and Excel formats.
Visual Studio Magazine
visualstudiomagazine.com › articles › 2017 › 12 › 01 › new-for-vs-code.aspx
New for VS Code: Support for JUnit, HDInsight PySpark, Python and R for Azure Data Lake -- Visual Studio Magazine
December 1, 2017 - "For PySpark developers who value productivity of Python language, VS Code HDInsight Tools offer you a quick Python editor with simple getting started experiences, and enable you to submit PySpark statements to HDInsight clusters with interactive responses," Jiang said. "This interactivity brings the best properties of Python and Spark to developers and empowers you to gain faster insights." ... Interactive responses brings the best properties of Python and Spark with flexibility to execute one or multiple statements · Built-in Python language service, such as IntelliSense auto-suggest, autocomplete and error marker, among others
Argon Systems
argonsys.com › home › msft articles › visual studio code: develop pyspark jobs for sql server 2019 big data clusters
Visual Studio Code: Develop PySpark jobs for SQL Server 2019 Big Data Clusters - Argon Systems
November 20, 2023 - Python editing: Develop PySpark applications with native Python authoring support (e.g. IntelliSense, auto format, error checking, etc.).
GitHub
github.com › microsoft › vscode-python › issues › 2628
PySpark setting for launch.json has bad default values · Issue #2628 · microsoft/vscode-python
September 19, 2018 - However I was still not able to get IntelliSense running as expected for PySpark. Turns out what happened was that the environment variable referencing was incorrect: The reference to SPARK_HOME should be ${env:SPARK_HOME} instead of ${env.SPARK_HOME} as specified in VSCode documentation
Published Sep 19, 2018
Analyticshut
analyticshut.com › integrate-spark-with-jupyter-notebook-and-visual-studio-code
Integrate Spark with Jupyter Notebook and Visual Studio Code | Analyticshut
November 24, 2024 - In last blog we had set up spark on our machine. We can access spark from console or command prompt. But when we are working, this is not the ideal way. It won’t save our commands, fixing errors in console is much difficult, and it does not have any intellisense.
Databricks
docs.gcp.databricks.com › developers › local development tools › databricks extension for visual studio code and cursor › command palette commands
Command Palette commands for the Databricks extension for Visual Studio Code | Databricks on Google Cloud
April 12, 2024 - This article describes how to enable PySpark and Databricks Utilities code completion for the Databricks extension for Visual Studio Code. See What is the Databricks extension for Visual Studio Code? This information assumes that you have already installed and set up the Databricks extension for Visual Studio Code. See Install the Databricks extension for Visual Studio Code. To enable IntelliSense (also known as code completion) in the Visual Studio Code code editor for PySpark, Databricks Utilities, and related globals such as spark and dbutils, do the following with your project opened:
Medium
nidhig631.medium.com › exploring-pyspark-setup-in-visual-studio-code-23f2bb778dde
Exploring PySpark Setup in Visual Studio Code | by Namaste Databricks | Medium
August 24, 2024 - This article provides a step-by-step guide to setting up your environment, leveraging the robust capabilities of PySpark, and seamlessly integrating it into the VS Code.
Visual Studio Code
code.visualstudio.com › docs › python › editing
Editing Python in Visual Studio Code
November 3, 2021 - Pylance is the default language server for Python in VS Code, and is installed alongside the Python extension to provide IntelliSense features.
Medium
medium.com › @vikassharma555 › vcode-create-pyspark-scripts-notebook-on-wsl2-ubuntu-on-windows-67c17a0d23e7
VCode create Pyspark Scripts/Notebook on WSL2 Ubuntu on Windows | by Vikas Sharma | Medium
April 26, 2022 - The red marks indicate extensions installed on local system in VSCode and extentions installed in WSL Ubuntu. Check options “Install in WSL:Ubuntu20.03). This indicates that extension is already installed in Windows locally but not installed on WSL. Install Python (for python script), Pylance(for intellisense…code completions)and Jupyter (for notebook) all from (from microsoft) under WSL section:
Microsoft Learn
learn.microsoft.com › en-us › sql › big-data-cluster › spark-hive-tools-vscode
Run jobs: Spark & Hive Tools for VS Code - SQL Server Big Data Clusters | Microsoft Learn
March 3, 2023 - You can find the Livy configuration by selecting settings.json under .vscode. ... For settings driverMemory and executorMemory, set the value with unit, for example 1gb or 1024mb. ... The created batch object. ... The assigned Livy config will display in output pane when submit script. Spark & Hive for Visual Studio Code supports the following features: IntelliSense autocomplete.
Kontext
kontext.tech › home › blogs › spark & pyspark › debug pyspark code in visual studio code
Debug PySpark Code in Visual Studio Code - Kontext Labs
January 4, 2020 - The page summarizes the steps required to run and debug PySpark (Spark for Python) in Visual Studio Code.
Top answer 1 of 2
1
The solution I found was to use the Jedi language server:-
"python.languageServer": "Jedi"
For Jupyter notebook you also need to set:-
"jupyter.enableExtendedPythonKernelCompletions": true
2 of 2
0
As mentioned in doc, you need to install and set up the Databricks extension for Visual Studio Code.
To enable IntelliSense (also known as code completion) in the Visual Studio Code code editor for PySpark, Databricks Utilities, and related globals such as spark and dbutils, do the following with your code project opened:
- On the Command Palette (View > Command Palette), type >Databricks: Configure autocomplete for Databricks globals and press Enter.
- Follow the on-screen prompts to allow the Databricks extension for
Visual Studio Code to install PySpark for your project, and to add
or modify the
__builtins__.pyifile for your project to enable Databricks Utilities.