Geek Answers Handbook

Spring + Maven + Hadoop

I have a problem in Spring + Maven + Hadoop environment. I am using Apache Hadoop but not using spring -hadoop which is provided from SpringSource. I added the following Maven dependency to use hadoop in java.

<dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-core</artifactId> <version>0.20.2</version> <exclusions> <exclusion> <groupId>org.mortbay.jetty</groupId> <artifactId>jetty</artifactId> </exclusion> <exclusion> <groupId>org.mortbay.jetty</groupId> <artifactId>jetty-util</artifactId> </exclusion> <exclusion> <groupId>org.mortbay.jetty</groupId> <artifactId>jsp-2.1</artifactId> </exclusion> <exclusion> <groupId>org.mortbay.jetty</groupId> <artifactId>jsp-api-2.1</artifactId> </exclusion> <exclusion> <groupId>org.mortbay.jetty</groupId> <artifactId>servlet-api-2.1</artifactId> </exclusion> <exclusion> <groupId>javax.servlet</groupId> <artifactId>servlet-api</artifactId> </exclusion> <exclusion> <groupId>javax.servlet.jsp</groupId> <artifactId>jsp-api</artifactId> </exclusion> <exclusion> <groupId>tomcat</groupId> <artifactId>jasper-compiler</artifactId> </exclusion> <exclusion> <groupId>tomcat</groupId> <artifactId>jasper-runtime</artifactId> </exclusion> </exclusions> </dependency>

And I did the work with the dooop map. Mapreduce work well.

But I can’t see the completed tasks from the "Launching tasks" list in the Hadoop web interface ( http://xxx.xxx.xxx.xxx//0030/jobtracker.jsp )

I started without the hadoop command (ex-bin / hadoop -jar .... jar), but just a java application. Could this be the reason?

+4
source share
1 answer

Hadoop provides an API for running MR jobs locally, without setting up a Hadoop cluster. If you do not see that your work is started / completed in the Hadoop web interface, it seems that you are doing the MR work locally.

To start remotely - specify the appropriate job tracking host, in your case http://xxx.xxx.xxx.xxx:9001

0
source

More articles:


All Articles