Geek Answers Handbook

How can I compile / use mahout for hadoop 2.0?

The latest version of mahout 0.9 is built only on hadoop 1.x. (mvn clean install) How can I compile mahout for hadoop 2.0.x?

Because when I ran the commands:

hasoop jar mahout-examples-0.9-SNAPSHOT-job.jar org.apache.mahout.cf.taste.hadoop.item.RecommenderJob -s SIMILARITY_COOCCURENCE -i test -o result

I always got an error message

IncompatibleClassChangeError: the org.apache.hadoop.mapreduce.JobContext interface was found, but the class was expected.

Thanks!

+4
source share
3 answers

Compile Mahout to work with 2.x, since it is not released in a package compatible with Hadoop 2.x:

mvn clean install -Dhadoop2 -Dhadoop2.version=2.2.0 -DskipTests=true

If you want to confirm after the build that the correct dependencies have been entered, do the following from the project root:

 find . -name hadoop*.jar

The artifacts generated by the above command are different from the artifacts in version 0.9, so you will need to update more than just the version number (there seems to be a new banner mahout-mrlegacy).

If someone can point me to an official artifact that works with 2.x (even if it's SNAPSHOT), I would appreciate it.

+1
source

Have you tried changing the hadoop version in pom.xml?

 <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-core</artifactId> <version> 0.23.9</version>

(Please check hadoop version. As I understand it, 0.23 became 2.0)

Another thing to try is to run the Recommender as follows:

 bin/mahout recommenditembased \ --input INPUT --output OUTPUT \ --tempDir TEMP --similarityClassname SIMILARITY_LOGLIKELIHOOD

To accomplish this, you must be at the root of your mahout folder, and you need to set the following environment variables:

 export HADOOP_HOME=/path/to/hadoop/home export HADOOP_COMMONS_HOME=/path/to/commons/home export JAVA_HOME=/path/to/java/home export PATH="$PATH:$HADOOP_HOME/bin"
0
source

About an hour ago, Mahout officially added Hadoop 2.x support to the main branch (see MAHOUT-1329 )

Checkout the code here https://github.com/apache/mahout and recompile with:

 mvn clean package -Dhadoop2.version=2.2.0

Try it and see if it works.

0
source

More articles:


All Articles