I am trying to use MRunit to test my sortComparatorClass . It seems MRunit should be able to do this using the setKeyOrderComparator method, but when I run mapReduceDriver , it does not call the compare() method of the SortComparator class.
Pretty sure I'm doing something wrong with the MRunit API.
Here is my code for unit test:
public class UnitTests { private static transient Log log = LogFactory.getLog(UnitTests.class); MapReduceDriver<Text, Text, Text, Text, Text, Text> mapReduceDriver; MapDriver<Text, Text, Text, Text> mapDriver; ReduceDriver<Text, Text, Text, Text> reduceDriver; @Before public void setUp() throws InterruptedException, IOException { mapDriver = new MapDriver<Text, Text, Text, Text>(); mapDriver.setMapper(new TestMapper()); reduceDriver = new ReduceDriver<Text, Text, Text, Text>(); reduceDriver.setReducer(new TestReducer()); mapReduceDriver = new MapReduceDriver(new TestMapper(), new TestReducer()); mapReduceDriver.setKeyOrderComparator(new TestSortCompartor()); } @Test public void testSort() throws IOException { Text inputKey1 = new Text("def"); Text inputKey2 = new Text("abc"); Text inputValue = new Text("BlahBlahBlah"); mapReduceDriver.addInput(inputKey1, inputValue); mapReduceDriver.addInput(inputKey2, inputValue); List<Pair<Text, Text>> output = mapReduceDriver.run(); log.info("Got output of size "+output.size()+" with first pair = "+output.get(0).toString()); } }
And here is my SortComparator test:
public class TestSortCompartor extends WritableComparator{ private static transient Log log = LogFactory.getLog(TestSortCompartor.class); public TestSortCompartor() { super(Text.class, true); } @SuppressWarnings("rawtypes") @Override public int compare(WritableComparable w1, WritableComparable w2) { log.info("calling compare with key1 = "+w1.toString()+" and key2 "+w2.toString()); return w1.compareTo(w2) * -1; } }
When I run the test, I get this output:
INFO 2014-01-13 09:34:27,362 [main] (com.gradientx.gxetl.testMR.UnitTests:53) - Got output of size 2 with first pair = (abc, BlahBlahBlah)
But there is no result from the SortComparator class - and it does not order the keys in the reverse order, so I know that my comparator() method is not called.
Can anyone advise what I am doing wrong? Can MRunit be used to test my own comparator class? Is there a better way to do a unit test for a custom comparator class?
FYI, here are the relevant dependencies on my Pom:
<dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-core</artifactId> <version>2.0.0-mr1-cdh4.4.0</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>2.0.0-mr1-cdh4.4.0</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.mrunit</groupId> <artifactId>mrunit</artifactId> <version>1.0.0</version> <classifier>hadoop2</classifier> </dependency>