I have sparking using checkpoint recording on HDFS.
Does anyone know a solution? Previously, we used kinit to specify the principal and keytab and received a proposal to specify them using the spark-submit kinit instead of kinit , but still this error led to the launch of the sparking application.
spark-submit --principal sparkuser@HADOOP.ABC.COM --keytab /home/sparkuser/keytab/sparkuser.keytab --name MyStreamingApp --master yarn-cluster --conf "spark.driver.extraJavaOptions=-XX:+UseConcMarkSweepGC --conf "spark.eventLog.enabled=true" --conf "spark.streaming.backpressure.enabled=true" --conf "spark.streaming.stopGracefullyOnShutdown=true" --conf "spark.executor.extraJavaOptions=-XX:+UseConcMarkSweepGC --class com.abc.DataProcessor myapp.jar
I see several cases of the following exception in the logs, and finally SIGTERM 15, which kills the artist and driver. We use CDH 5.5.2
2016-10-02 23:59:50 ERROR SparkListenerBus LiveListenerBus: 96 - Listener EventLoggingListener throws an exception java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor8.invoke (Unknown source) at sun.reflect.Delegatingatingethethpleth. java: 43) in java.lang.reflect.Method.invoke (Method.java:606) at org.apache.spark.scheduler.EventLoggingListener $$ anonfun $ logEvent $ 3.apply (EventLoggingListener.scala: 148) at org.apache .spark.scheduler.EventLoggingListener $$ anonfun $ logEvent $ 3.apply (EventLoggingListener.scala: 148) when scala.Option.foreach (Option.scala: 236) on org.apache.spark.scheduler.EventLoggingListener.logEvent (EventLoggingListener : 148) at org.apache.spark.scheduler.EventLoggingListener.onUnpersistRDD (EventLoggingListener.scala: 184) at org.apache.spark.scheduler.SparkListenerBus $ class.onPostEvent (SparkListenerBus.scala: 50) at org.apache.spark.scheduler.LiveListenerBus.onPostEvent (LiveListenerBus.scala: 31) at org.apache.spark.scheduler. .onPostEvent (LiveListenerBus.scala: 31) at org.apache.spark.util.ListenerBus $ class.postToAll (ListenerBus.scala: 56) at org.apache.spark.util.AsynchronousListenerBus.postToAll (AsynchronousListenerBus.scala: org.apache.spark.util.AsynchronousListenerBus $$ anon $ 1 $$ anonfun $ run $ 1.apply $ mcV $ sp (AsynchronousListenerBus.scala: 79) at org.apache.spark.util.Utils $ .tryOrStopSparkContext (Utils.scala: 1135) at org.apache.spark.util.AsynchronousListenerBus $$ anon $ 1.run (AsynchronousListenerBus.scala: 63) Called: java.io.IOException: rental period 0 seconds expired. at org.apache.hadoop.hdfs.DFSOutputStream.abort (DFSOutputStream.java:2370) at org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten (DFSClient.java:964) at org.apache.hadoop.hdfs.DLerenLeSClfrenlfes (DFSClient.java:932) at org.apache.hadoop.hdfs.LeaseRenewer.renew (LeaseRenewer.java:423) at org.apache.hadoop.hdfs.LeaseRenewer.run (LeaseRenewer.java:448) at org.apache. hadoop.hdfs.LeaseRenewer.access $ 700 (LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer $ 1.run (LeaseRenewer.java:304) in java.lang.Thread.run (Thread.java:745)
source share