Geek Answers Handbook

Error handling in hadoop map reduces

Based on the documentation, there are several ways that error handling is performed on the map. Below are a few:

and. Custom counters using enum - increment for each failed record.

b. Log error and analysis later.

Counters give the number of failed records. However, in order to get the identifier of the failed record (it may be its unique key), as well as information about the exception, the node on which the error occurred, we need to perform a centralized analysis of the log, and there are many running nodes. Logstash available.

In addition, are there other ways to handle error scenarios without manual intervention. Any tools, links, best practices are welcome.

I think the same method applies to any distributed applications with minor changes.

+3
source share
1 answer

A few questions to ask when dealing with error handling:

  • If the task is stopped, if an error occurred while checking the data. Most cases of using big data may be in order to leave few bad records. But if your usecase wants all the entries to be good enough, you must make that decision and go to the steps below.

    , , (), , .

  • , , . , , , ,

    : mapreduce.map.maxattempts

    : mapreduce.reduce.maxattempts

    4

  • .

    , . . , .

    Mapper

    enum Temperature { OVER_10 }

    ,

    //

    if(value > 10) {
    System.err.println("Temperature over 100 degrees for input: " + value);
    context.setStatus("Detected possibly corrupt record: see logs.");
    context.getCounter(Temperature.OVER_10).increment(1);      
}

    , . , - .

    $mapred job -counter <job_id> '${fully_qualified_class_name}' ${enum_name}
$mapred job -counter job_1444655904448_17959 'com.YourMapper$Temperature' OVER_10

    , " ". .

    hdf.

    yarn logs -applicationId <application ID>
+1

More articles:


All Articles