Debugging a distributed application is still challenging in the notebook environment. Even though the web UI has the necessary information, there is a gap between web UIs and the development environment: it’s usually difficult to locate information in the web UI that is relevant to the code you are investigating; and there is no easy way to find historical runtime information.
Understanding how to debug with the Databricks Spark UI:
The Spark UI contains a wealth of information you can use for debugging your Spark jobs. There are a bunch of great visualizations, and we have a blog post here about those features.
For more details, click on Jobx View (Stages):
Reference: Tips to Debug Apache Spark UI with Databricks
Hope this helps.
Answer from CHEEKATLAPRADEEP on Stack OverflowDatabricks
databricks.com › blog › 2016 › 10 › 18 › 7-tips-to-debug-apache-spark-code-faster-with-databricks.html
7 Tips to Debug Apache Spark Code Faster with Databricks
Therefore, you'll want to factor your code such that you can store intermediary RDDs / Dataframes as a variable. When debugging, you should call count() on your RDDs / Dataframes to see what stage your error occurred.
Microsoft Learn
learn.microsoft.com › en-us › azure › databricks › notebooks › debugger
Debug notebooks - Azure Databricks | Microsoft Learn
Clicking Inspect automatically executes code in the debug console to output the value of your variable. To filter the display, enter text into the search box. The list is automatically filtered as you type.
How to interactively debug a Python wheel in a Databricks Asset Bundle?
Is the library yours? Any whls I’ve created and wanted to do what u ask I’d download the source code and run that in debug. More on reddit.com
12
6
July 2, 2025python - databricks - how to debug parameters when calling function - Stack Overflow
In databricks, I have 2 notebooks. One notebooks holds a function. The other notebook calls the function in the first notebook. this is how I am calling the function from pyspark.sql.functions im... More on stackoverflow.com
apache spark - Using databricks-connect debugging a notebook that runs another notebook - Stack Overflow
I am able to connect to the Azure Databricks cluster from my Linux Centos VM, using visual studio code. Below code even works without any issue from pyspark.sql import SparkSession spark = SparkSe... More on stackoverflow.com
PySpark style guide
Thank you this is great. Always thought that palantir style guide was dreadful. You have converged to a lot of similar patterns as I have, which is reassuring. Main difference is I somehow missed that spark 3.3 lets you pass additional args to transform functions so I'd been using the inner function approach to achieve the same effect, much cleaner to do it the new way though! More on reddit.com
3
10
January 18, 2023Videos
11:14
Connect; 2017 Breakpoint debugging of Spark jobs in Azure HDInsight ...
47:14
Tuning and Debugging Apache Spark - YouTube
25:58
Debugging PySpark -- Or trying to make sense of a JVM stack trace ...
27:32
Debugging PySpark: Spark Summit East talk by Holden Karau - YouTube
40:00
Holden Karau | Debugging PySpark - Pretending to make sense of ...
Databricks
databricks.com › blog › 2015 › 09 › 23 › easier-spark-code-debugging-real-time-progress-bar-and-apache-spark-web-ui-integration-in-databricks.html
Easier Spark Code Debugging | Databricks Blog
By integrating Spark's web UI with Databricks notebooks, we have created a shortcut to easily access debugging information within your development environment. Hopefully these enhancements will help you debug Spark application more effectively. These enhancements are now available to all Databricks users, sign up for a 14-day free trial to try them out!
Pragmatic Works
pragmaticworks.com › blog › debug-and-write-pyspark-code-with-the-ai-assistant-in-databricks
Debug and Write PySpark Code with the AI Assistant in Databricks
In this tutorial, Mitchell Pearson demonstrates how to leverage the AI Assistant in Databricks to enhance your PySpark coding experience. This powerful tool can help you debug code and even generate PySpark scripts, making it a must-have feature for any data professional working in Databricks
Databricks
docs.databricks.com › local development tools › databricks extension for visual studio code and cursor › debug notebooks with databricks connect
Run and debug notebook cells with Databricks Connect using the Databricks extension for Visual Studio Code | Databricks on AWS
January 19, 2026 - spark, representing an instance of databricks.connect.DatabricksSession, is preconfigured to instantiate DatabricksSession by getting Databricks authentication credentials from the extension. If DatabricksSession is already instantiated in a notebook cell's code, this DatabricksSession settings are used instead. See Code examples for Databricks Connect for Python. udf, preconfigured as an alias for pyspark.sql.functions.udf, which is an alias for Python UDFs.
Databricks
docs.databricks.com › compute › classic compute › troubleshoot compute issues › debugging with the spark ui
Debugging with the Spark UI | Databricks on AWS
In the Executors table, find the row that contains the Executor ID value that corresponds to the Executor ID value that you noted earlier. In that row, click the link in the Thread Dump column.
Databricks Documentation
docs.databricks.com › notebooks › develop code in notebooks › debug notebooks
Debug notebooks | Databricks on AWS
January 29, 2025 - Learn how to use the Databricks interactive debugger for Python notebooks.
Databricks
databricks.com › blog › new-debugging-features-databricks-notebooks-variable-explorer
New Debugging Features in Databricks | Databricks Blog
May 17, 2023 - View your notebook's variable state, inspect and explore data frames in one click, and interactively debug Python code with pdb and the new Variable Explorer.
Reddit
reddit.com › r/databricks › how to interactively debug a python wheel in a databricks asset bundle?
r/databricks on Reddit: How to interactively debug a Python wheel in a Databricks Asset Bundle?
July 2, 2025 -
Hey everyone,
I’m using a Databricks Asset Bundle deployed via a Python wheel.
Edit: the library is in my repo and mine, but quite complex with lots of classes so I cannot just copy all code in a single script but need to import.
I’d like to debug it interactively in VS Code with real Databricks data instead of just local simulation.
Currently, I can run scripts from VS Code that deploy to Databricks using the vscode extension, but I can’t set breakpoints in the functions from the wheel.
Has anyone successfully managed to debug a Python wheel interactively with Databricks data in VS Code? Any tips would be greatly appreciated!
Edit: It seems my mistake was not installing my library in the environment I run locally with databricks-connect. So far I am progressing, but still running in issues when loading files in my repo which is usually in workspace/shared. Guess I need to use importlib to get this working seamlessly. Also I am using some spark attributes that are not available in the connect session, which require some rework. So to early to tell if in the end I am succesful, but thanks for the input so far.
Thanks!
Top answer 1 of 4
3
Is the library yours? Any whls I’ve created and wanted to do what u ask I’d download the source code and run that in debug.
2 of 4
3
If you’re trying to debug a python wheel from a databricks asset bundle in vs code with real databricks data, here’s a practical way to do it: Use databricks connect v2 – set it up with the same python and spark versions as your cluster so everything runs smoothly. Install your library locally – use pip install -e . so you can set breakpoints and step through the actual source code. Set up vs code for debugging – create a launch.json and point it to a .env file with your databricks config. this lets you run and debug like it’s local, but on remote data. Avoid __main__ logic – move your main logic into functions so they’re easier to test and debug. Access workspace files properly – files in dbfs:/workspace/... should be read using dbutils.fs or the /dbfs/... path. Handle unsupported apis – some spark features won’t work with connect. wrap them so you can mock or bypass when needed.
Databricks
databricks.com › blog › debug-your-code-and-notebooks-using-visual-studio-code
Debug Code & Notebooks in VS Code | Databricks Blog
June 12, 2023 - Databricks Connect enables running ... to use Databricks Connect (see "Easy Setup" below), simply use the VS Code "Debug Python File" button to start debugging....
Databricks Documentation
docs.databricks.com › developers › local development tools › databricks extension for visual studio code and cursor › run tests
Run Python tests using the Databricks extension for Visual Studio Code | Databricks on Google Cloud
January 16, 2026 - This example assumes that this file is named spark_test.py and is at the root of your Visual Studio Code project. This file contains a pytest fixture, which makes the cluster's SparkSession (the entry point to Spark functionality on the cluster) available to the tests. This file contains a single test that checks whether the specified cell in the table contains the specified value. You can add your own tests to this file as needed. ... from pyspark.sql import SparkSession import pytest @pytest.fixture def spark() -> SparkSession: # Create a SparkSession (the entry point to Spark functionality) on # the cluster in the remote Databricks workspace.
Stack Overflow
stackoverflow.com › questions › 77929554 › databricks-how-to-debug-parameters-when-calling-function
python - databricks - how to debug parameters when calling function - Stack Overflow
you have to use User Defined Functions (UDFs) to convert the Python function to a Pyspark function, refer: changhsinlee.com/pyspark-udf and walkenho.github.io/…
Medium
medium.com › @gema.correa › debugging-notebooks-in-databricks-4d3543f1ee7e
Debugging Notebooks in Databricks | by Gema Correa | Dec, 2025 | Medium
December 8, 2025 - This post walks you through how to use the new debugger, what nobody tells you about it, and several practical examples you can use today. ... Databricks often needs a fresh interpreter before running its first debug session.
Databricks
docs.gcp.databricks.com › en › dev-tools › vscode-ext › dev-tasks › databricks-connect.html
Debug code by using Databricks Connect for the Databricks extension for Visual Studio Code | Databricks on Google Cloud
Then in the drop-down list, select Debug Python File. This choice supports step-through debugging, breakpoints, watch expressions, call stacks, and similar features. This choice uses Databricks Connect to run Python code locally, run PySpark code on the cluster in the remote workspace, and send remote responses back to Visual Studio Code for local debugging.
Databricks
databricks.com › session › debugging-pyspark-or-why-is-there-a-jvm-stack-trace-in-my-python
Databricks - YouTube
October 18, 2016 - Databricks is the Data and AI company. More than 20,000 organizations worldwide — including adidas, AT&T, Bayer, Block, Mastercard, Rivian, Unilever, and over 60% of the Fortune 500 — rely on Databricks to build and scale data and AI apps, analytics and agents.
Microsoft Learn
learn.microsoft.com › en-us › azure › databricks › clusters › debugging-spark-ui
Debugging with the Spark UI - Azure Databricks | Microsoft Learn
May 3, 2023 - Learn how to troubleshoot and debug Spark applications using the UI and compute logs in Azure Databricks.
Databricks Documentation
docs.databricks.com › notebooks › develop code in notebooks › debug notebooks
Debug notebooks | Databricks on Google Cloud
Learn how to use the Databricks interactive debugger for Python notebooks.