Geek Answers Handbook

The difference in failed tasks versus dead tasks

From the Jobtracker user interface, I see this column called "Failed / Killed Tasks".

I would like to know the difference between the two. I think that “Failed” means tasks that really failed in the end after some attempts (so there was no recovery at all?), While “Killed” means tasks that kill (due to timeout and t .d.) But can they be repeated?

+8
mapreduce hadoop
source share
5 answers

There are several reasons why Hadoop can kill tasks with its solutions:
a) The task does not report progress during a timeout (10 minutes by default)
b) FairScheduler or CapacityScheduler needs a slot for another pool (FairScheduler) or queue (CapacityScheduler).
c) Speculative execution leads to the fact that the results of the task are not required because they are completed elsewhere.

+11
source share

Hadoop uses speculative execution. The same task can be run in multiple boxes. The first wins, and the remaining copies are killed.

Unsuccessful tasks are tasks that exit the system.

+7
source share

I would not agree with David a bit, because the timeout is not marked as dead, but instead marked as unsuccessful.

In my understanding, below are the definitions of failed vs killed tasks

Task may be crashing due

  • task throws an exception at run time
  • sudden exit of a child JVM
  • timeout exceeds mapred.task.timeout

Mission may be killed due to

  • For FairScheduler or CapacityScheduler, you need a slot for another pool (FairScheduler) or queue (CapacityScheduler).
  • Speculative execution leads to the fact that the results of the task are not needed, because they are completed elsewhere.
  • Users can also kill or not attempt tasks using the web interface or command line.
+1
source share

A Unsuccessful task is an attempt that caused an exception at runtime; a killed task is a duplicated task killed during speculative execution.

0
source share

Unsuccessful tasks 1. Buggy map or reduce code (exception error at runtime) 2. Sudden exit from jvm 3. Hanging task (this can cause speculative execution if it is enabled)

Killed tasks 1. Speculative execution kills long work. 2. If the user kills the job. 3. Task tracking error

0
source share

More articles:


All Articles