Geek Answers Handbook

How to enable spark history server for offline cluster mode without hdfs

I have a Spark2.1.1 cluster installed (1 leading 2 slave) after http://paxcel.net/blog/how-to-setup-apache-spark-standalone-cluster-on-multiple-machine/ offline. I do not have Hadoop pre-installed on the machine. I wanted to start a spark server. I run it as follows:

roshan@bolt :~/spark/spark_home/sbin$ ./start-history-server.sh

and in the spark-defaults.conf parameter I set this:

 spark.eventLog.enabled true

But the error fails:

 7/06/29 22:59:03 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(roshan); groups with view permissions: Set(); users with modify permissions: Set(roshan); groups with modify permissions: Set() 17/06/29 22:59:03 INFO FsHistoryProvider: History server ui acls disabled; users with admin permissions: ; groups with admin permissions Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:278) at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala) Caused by: java.io.FileNotFoundException: Log directory specified does not exist: file:/tmp/spark-events Did you configure the correct one through spark.history.fs.logDirectory? at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:214)

What should I install in spark.history.fs.logDirectory and spark.eventLog.dir

Update 1:

 spark.eventLog.enabled true spark.history.fs.logDirectory file:////home/roshan/spark/spark_home/logs spark.eventLog.dir file:////home/roshan/spark/spark_home/logs

but I always get this error:

 java.lang.IllegalArgumentException: Codec [1] is not available. Consider setting spark.io.compression.codec=snappy at org.apache.spark.io.Co
0
source share
2 answers

By default, spark defines file:/tmp/spark-events as the log directory for the history server, and your log clearly says that spark.history.fs.logDirectory is not configured

First of all, you need to create a spark events folder in / tmp (which is not very good, since / tmp is updated every time the machine reboots), and then add spark.history.fs.logDirectory to the default spark values. conf to point to this directory. But I suggest you create another folder in which the user can access and update the spark-defaults.conf file.

You need to define two more variables in the spark-defaults.conf file

 spark.eventLog.dir file:path to where you want to store your logs spark.history.fs.logDirectory file:same path as above

Suppose you want to save in / opt / spark -events, where the spark user has access to more specified parameters in spark-defaults.conf, will be

 spark.eventLog.enabled true spark.eventLog.dir file:/opt/spark-events spark.history.fs.logDirectory file:/opt/spark-events

You can find more information in Monitoring and Tools

0
source

Try to install

 spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec

in spark-defaults.conf

0
source

More articles:


All Articles