Geek Answers Handbook

Failed tasks resubmitted to Apache Spark?

Failed tasks are automatically resubmitted to Apache Spark to the same or different worker?

+4
source share
2 answers

I believe that failed tasks are resubmitted because I saw the same failed task presented several times in the web interface. However, if the same task fails several times, the job will fail completely:

org.apache.spark.SparkException: Job aborted due to stage failure: Task 120 in stage 91.0 failed 4 times, most recent failure: Lost task 120.3 in stage 91.0
+5
source

Yes, but there is a set of parameters for the maximum number of failures

spark.task.maxFailures  4   Number of individual task failures before giving up on the job. Should be greater than or equal to 1. Number of allowed retries = this value - 1.
+2
source

More articles:


All Articles