Hadoop MRUnit exception excludes

Question

I am trying to write some unit tests for my MR job and get the following exception. This is the first time I'm using MRUnit, so I'm not quite sure what is going on here.

java.lang.IncompatibleClassChangeError: Found class org.apache.hadoop.mapreduce.TaskInputOutputContext, but interface was expected at org.apache.hadoop.mrunit.mapreduce.mock.MockContextWrapper.createCommon(MockContextWrapper.java:53) at org.apache.hadoop.mrunit.mapreduce.mock.MockMapContextWrapper.create(MockMapContextWrapper.java:70) at org.apache.hadoop.mrunit.mapreduce.mock.MockMapContextWrapper.<init>(MockMapContextWrapper.java:62) at org.apache.hadoop.mrunit.mapreduce.MapDriver.run(MapDriver.java:217) at org.apache.hadoop.mrunit.MapDriverBase.runTest(MapDriverBase.java:150) at org.apache.hadoop.mrunit.TestDriver.runTest(TestDriver.java:137) 

My actual code looks very simple

 private MapDriver<Text, Text, Text, Text> mapDriver = MapDriver.newMapDriver(mapper); private ReduceDriver<Text, Text, Text, Text> reduceDriver = ReduceDriver.newReduceDriver(reducer); private MapReduceDriver<Text, Text, Text, Text, Text, Text> driver = MapReduceDriver.newMapReduceDriver(mapper, reducer); @Test public void testMapper() { mapDriver.withInput(new Text("1"), new Text("Line 1")); mapDriver.withOutput(new Text("1"), new Text("Line 1")); mapDriver.runTest(); } 

Answer 1

As you can see in here , in mrunit 0.9 there are two classifiers. And there are some problems using mrunit in Java 7 environment.

Perhaps you are debuting with hadoop 1.0.x and only added one of the two classifiers to your pom.xml file. The solution is easy: Just add another one as described below. What all.

 <dependencies> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-core</artifactId> <version>1.0.3</version> </dependency> <dependency> <groupId>org.apache.mrunit</groupId> <artifactId>mrunit</artifactId> <version>0.9.0-incubating</version> <classifier>hadoop1</classifier> </dependency> <dependency> <groupId>org.apache.mrunit</groupId> <artifactId>mrunit</artifactId> <version>0.9.0-incubating</version> <classifier>hadoop2</classifier> </dependency> </dependencies> 

I think this error will be resolved in the next release, but I cannot be sure.

Answer 2

This error tends to occur if the version of MRUnit that you use was created against a different version of Hadoop, and not the code that you are testing. In this particular case, I believe that TaskInputOutputContext has been changed from an abstract class to an interface in 0.21.

So, maybe you are using 0.21, and the version of MRUnit you are using is built against 0.20. I would just try to figure out the versions.

Answer 3

MRUnit has two release packages for Hadoop 1.x and Hadoop 2.x. You must make sure that the version of mrunit is equal to the version of hadoop.