What is the difference between spark.eventLog.dir and spark.history.fs.logDirectory?

Question

In Spark, what is the difference between an event log directory and a history server log directory?

spark.eventLog.dir hdfs:///var/log/spark/apps spark.history.fs.logDirectory hdfs:///var/log/spark/apps 

Answer 1

From the official Apache Spark documentation:

spark.eventLog.dir is the base directory in which Spark events are logged if true.eventLog.enabled is true. In this base directory, Spark creates a subdirectory for each application and logs application-specific events in this directory. Users may want to install this in a single location, for example, in the HDFS directory, so that historical files can be read by the history server.

See spark.eventLog.dir .

spark.history.fs.logDirectory is for the file system history provider, the URL of the directory into which application event logs are loaded. This can be a local file: // path, HDFS path hdfs: // namenode / shared / spark-logs or an alternative file system file supported by the Hadoop API.

See spark.history.fs.logDirectory .

spark.eventLog.dir generates logs, and spark.history.fs.logDirectory is where Spark History Server detects log events.