http://spark.apache.org/docs/latest/configuration.html#configuring-logging
Configuring Logging
Spark uses log4j for logging. You can configure it by adding a log4j.properties file in the conf directory. One way to start is to copy the existing log4j.properties.template located there.
The following blog about "How to log in spark" https://www.mapr.com/blog/how-log-apache-spark suggest a way to configure log4j, and provide suggestion which includes directing INFO level logs into a file.
Answer from Yaron on Stack Overflow Top answer 1 of 3
10
http://spark.apache.org/docs/latest/configuration.html#configuring-logging
Configuring Logging
Spark uses log4j for logging. You can configure it by adding a log4j.properties file in the conf directory. One way to start is to copy the existing log4j.properties.template located there.
The following blog about "How to log in spark" https://www.mapr.com/blog/how-log-apache-spark suggest a way to configure log4j, and provide suggestion which includes directing INFO level logs into a file.
2 of 3
10
you can also update the log level programmatically like below, get hold of spark object from JVM and do like below
def update_spark_log_level(self, log_level='info'):
self.spark.sparkContext.setLogLevel(log_level)
log4j = self.spark._jvm.org.apache.log4j
logger = log4j.LogManager.getLogger("my custom Log Level")
return logger;
use:
logger = update_spark_log_level('debug')
logger.info('you log message')
feel free to comment if you need more details
Medium
medium.com › @krishnatej2207 › debugging-and-logging-in-apache-spark-️-e90cf6cdda99
Debugging and Logging in Apache Spark 🔍🛠️ | by Krishnatej Sreeramula | Medium
January 11, 2025 - Spark uses the Log4j framework for logging, which provides flexibility for log management. You can configure logging levels and output formats to suit your debugging needs.
Spark By {Examples}
sparkbyexamples.com › home › apache spark › spark – stop info & debug message logging to console?
Spark - Stop INFO & DEBUG message logging to console? - Spark By {Examples}
March 27, 2024 - Problem: In Spark, wondering how to stop/disable/turn off INFO and DEBUG message logging to Spark console, when I run a Spark or PySpark program on a
Apache
spark.apache.org › docs › latest › api › python › reference › api › pyspark.SparkContext.setLogLevel.html
pyspark.SparkContext.setLogLevel - Apache Spark
SparkContext.setLogLevel(logLevel)[source]# Control our logLevel. This overrides any user-defined log settings. Valid log levels include: ALL, DEBUG, ERROR, FATAL, INFO, OFF, TRACE, WARN · New in version 1.4.0. Parameters · logLevelstr · The desired log level as a string.
Apache Spark
spark.apache.org › docs › latest › configuration.html
Configuration - Spark 4.1.1 Documentation
Spark uses log4j for logging. You can configure it by adding a log4j2.properties file in the conf directory.
The Internals of Spark SQL
jaceklaskowski.gitbooks.io › mastering-spark-sql › content › spark-logging.html
Logging · The Internals of Spark SQL
When running a Spark application from within sbt using run task, you can use the following build.sbt to configure logging levels: fork in run := true javaOptions in run ++= Seq( "-Dlog4j.debug=true", "-Dlog4j.configuration=log4j.properties") outputStrategy := Some(StdoutOutput) With the above configuration log4j.properties file should be on CLASSPATH which can be in src/main/resources directory (that is included in CLASSPATH by default).
Cloudera
docs.cloudera.com › cdsw › 1.10.5 › spark-configuration › topics › cdsw-spark-logging-configuration.html
Spark Logging Configuration
By default, a log4j.properties ... existing Spark logging properties for every session and job. To specify a custom location, set the environmental variable LOG4J_CONFIG to the file location relative to your project. The Log4j documentation has more details on logging options. Increasing the log level or pushing logs to an alternate location for troublesome jobs can be very helpful for debugging...
Databricks
docs.databricks.com › compute › classic compute › troubleshoot compute issues › debugging with the spark ui
Debugging with the Spark UI | Databricks on AWS
Learn how to troubleshoot and debug Spark applications using the UI and compute logs in Databricks.
Cloudera
docs.cloudera.com › cdp-private-cloud-base › 7.1.8 › monitoring-and-diagnostics › topics › cm-viewing-and-debugging-spark-applications-using-logs.html
Viewing and Debugging Spark Applications Using Logs
To debug Spark applications running on YARN, view the logs for the NodeManager role.
Databricks
kb.databricks.com › clusters › set-executor-log-level.html
Set executor log level - Databricks
July 19, 2022 - %scala sc.parallelize(Seq("")).foreachPartition(x => { import org.apache.log4j.{LogManager, Level} import org.apache.commons.logging.LogFactory LogManager.getRootLogger().setLevel(Level.DEBUG) val log = LogFactory.getLog("EXECUTOR-LOG:") log.debug("START EXECUTOR DEBUG LOG LEVEL") }) To verify that the level is set, navigate to the Spark UI, select the Executors tab, and open the stderr log for any executor:
Medium
medium.com › @dlnavgare › best-practices-to-debug-spark-applications-70ffb5964cdc
best practices to debug Spark applications | by Data Dnyan | Medium
July 18, 2023 - Debugging Spark applications can ... are some best practices to help you effectively debug Spark applications: Enable logging and adjust log levels: Configure Spark to log relevant ......
Cloudera Community
community.cloudera.com › t5 › Support-Questions › How-to-launch-spark-shell-in-debug-mode › td-p › 199418
Solved: How to launch spark-shell in debug mode - Cloudera Community - 199418
September 16, 2022 - Then run the spark-shell as following then you should see DEBUG messages. # su - spark # spark-shell --master yarn --deploy-mode client --files /tmp/log4j.properties --conf "spark.executor.extraJavaOptions='-Dlog4j.configuration=log4j.properties'" --driver-java-options "-Dlog4j.configuration=file:/tmp/log4j.properties" Multiple versions of Spark are installed but SPARK_MAJOR_VERSION is not set Spark1 will be picked by default 2018-09-17 07:52:29,343 [main] DEBUG org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.Us
GitHub
gist.github.com › czxm › 570f3b50515f81d27abcfb9d3a550a8c
Enable Spark DEBUG Logging · GitHub
Enable Spark DEBUG Logging. GitHub Gist: instantly share code, notes, and snippets.
Spark Code Hub
sparkcodehub.com › spark-how-to-debug-spark-applications
Debugging Spark Applications: Diagnose and Fix Issues Like a Pro
Learn how to debug Spark applications with the Spark UI logs and execution plans Resolve performance memory and logic issues with Scala and PySpark examples
Open Source at AWS
aws.github.io › aws-emr-containers-best-practices › troubleshooting › docs › change-log-level
Change Log Level - EMR Containers Best Practices Guides
In this example log4j.properties is placed in a s3 bucket that is mapped to a FSx for Lustre filesystem. Request pi.py used in the below request payload is from spark examples · cat > Spark-Python-in-s3-debug-log.json << EOF { "name": "spark-python-in-s3-debug-log", "virtualClusterId": "<virtual-cluster-id>", "executionRoleArn": "<execution-role-arn>", "releaseLabel": "emr-6.2.0-latest", "jobDriver": { "sparkSubmitJobDriver": { "entryPoint": "s3://<s3 prefix>/pi.py", "sparkSubmitParameters": "--conf spark.driver.cores=2 --conf spark.executor.memory=2G --conf spark.driver.memory=2G --conf spar
Spark Code Hub
sparkcodehub.com › spark › configurations › log-level
Mastering Apache Spark’s Logging Configuration: A Comprehensive Guide to spark.logConf and Log Levels
Description: Sets the root logger’s level for Spark and application logs. ... Level.OFF). Returns: None. ... TRACE: Most verbose, includes all events (rarely used). DEBUG: Detailed diagnostics (e.g., task execution, shuffle details).
Medium
medium.com › @sksami1997 › logging-in-spark-making-sense-of-the-chaos-15a0dc2f5212
Logging in Spark: Making Sense of the Chaos | by Shaik Sameer | Medium
March 5, 2025 - WARN — Logs potential issues. INFO — General operational messages (default level in Spark). DEBUG — Detailed logs useful for troubleshooting.
Microsoft Learn
learn.microsoft.com › en-us › azure › databricks › compute › troubleshooting › debugging-spark-ui
Debugging with the Spark UI - Azure Databricks | Microsoft Learn
October 7, 2025 - Learn how to troubleshoot and debug Spark applications using the UI and compute logs in Azure Databricks.
IBM
cloud.ibm.com › docs › AnalyticsEngine
Configuring Spark log level information
Find documentation, API & SDK references, tutorials, FAQs, and more resources for IBM Cloud products and services.
IBM
ibm.com › docs › en › cloud-paks › cp-data › 4.8.x
Configuring Spark log level information
4.8.1 and later Review the applications that run and identify the issues that are present by using the logs that the Analytics Engine powered by Apache Spark Spark application generates. The standard logging levels available are ALL, TRACE, DEBUG, INFO, WARN, ERROR, FATAL, and OFF.