Geek Answers Handbook

How to run Hadoop with a Java class?

I follow the Hadoop book : The Ultimate Guide .

I got confused in example 3-1.

There is a Java source file, URLCat.java. I use javac to compile it into URLCat.class, then use jar to wrap it in a jar.

The book says that

 % hadoop URLCat hdfs://localhost/user/tom/quangle.txt

to run it. I tried many different ways, for example

 % hadoop jar URLCat.jar .......

but does not work. I got the following errors:

Exception in thread "main" java.lang.ClassNotFoundException: hdfs: // localhost / user / username / quangle / txt

What is the reason for this and how can I do it right?

+4
source share
9 answers

The command syntax is slightly different:

 hadoop fs -cat hdfs:///user/tom/quangle.txt

Do you have a home on your way? can you call hadoop without any parameters?

0
source

It is pretty simple:

 [ me@myhost ~]$ hadoop jar RunJar jarFile [mainClass] args...

So you want hadoop jar yourJar.jar your.class.with.Main [any args]

+4
source

Of course, you can use cat, but it’s not (that is, you are learning, not just trying to make it work).

According to the book, you need to set the environment variable HADOOP_CLASSPATH . In my case, using the build example in the book, all my classes are: / media / data / hadefguide / book / build / classes

Here is an example:

 hduser@MuleBox ~ $ export HADOOP_CLASSPATH= hduser@MuleBox ~ $ hadoop URLCat hdfs://localhost/user/hduser/quangle.txt Exception in thread "main" java.lang.NoClassDefFoundError: URLCat Caused by: java.lang.ClassNotFoundException: URLCat at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) Could not find the main class: URLCat. Program will exit. hduser@MuleBox ~ $ export HADOOP_CLASSPATH=/media/data/hadefguide/book/build/classes hduser@MuleBox ~ $ hadoop URLCat hdfs://localhost/user/hduser/quangle.txt On the top of the Crumpetty Tree The Quangle Wangle sat, But his face you could not see, On account of his Beaver Hat.
+3
source

To make the hadoop URLCat command work, you need the jar (URLCat.jar) to be in your class path. You can put it in lib / dir hadoop for this.

To run runoap jar URLCat.jar, you need to create a jar in which the Main class will be defined, otherwise it considers that the next argument on the command line is the class name. What you can try is hasoop jar URLCat.jar URLCat hdfs: // ...

0
source

I did this based on the help found on this site and the tutorial.

 mkdir urlcat_classes<br> javac -classpath /usr/lib/hadoop/hadoop-0.20.2-cdh3u1-core.jar -d urlcat_classes URLCat.java<br> jar -cvf urlcat.jar -C urlcat_classes .<br> hadoop jar urlcat.jar no.gnome.URLCat hdfs://localhost/user/claus/sample.txt<br> <br> no.gnome is from 'package no.gnome;' in URLCat.java.<br><br>

respectfully
Klaus

0
source

Not sure how useful the answer is. Today I faced the same problem, actually working on an example from the same book (the final Hadoop manual) I was able to execute the sample program as follows:

  • Write your Java code and save it as a .java file

  • Compile your Java program using:

     javac -classpath <path to hadoop core and commons-cli jar file> <path to your java program file>

  • Create a jar file containing your class file:

     jar cvf <jar file> <class files to add separated by space>

  • Execute the jar file using the hadoop command line:

     hadoop jar <jar file name> <class name containing your main method> <argument to the main method>

    eg

     hadoop jar FileSystemCat.jar FileSystemCat hdfs://localhost/user/root/MyFiles/meet_a_seer.txt

Hope this helps

0
source

step 1: Compile the Java program:

javac URLCat.java -classpath $ HADOOP_HOME / share / hadoop / common / hadoop-common-2.7.0.jar

Step 2: Create a jar file:

jar cvf URLCat.jar URLCat.class

Step 3: Run the program: (specify the location of the hdfs file)

hasoop jar URLCat.jar URLCat hdfs: // localhost: 9000 / pcode / wcinput.txt

0
source

Change to the directory where your compiled .class files reside.

Use the fully qualified class name, including the package name (see Getting and the wrong name "NoClassDefFoundError when running a Java program from the command line for the full class name or the directory to start the job) when running hadoop URLCat hdfs://localhost/user/tom/quangle.txt .

In my case, URLCat.java was in com.tom.app , so the hadoop command was hadoop com.tom.app.URLCat hdfs://localhost/user/tom/quangle.txt .

0
source

We can access HDFS via hdfs api. My understanding of this is that you can use hdfs api to contact the hadoop cluster, which runs dfs and extracts data from it.

Why do we need to call the command as hasoop jar URLCat.jar

why not just java URLCat

Why does the client need to install hadoop and then contact the hadoop cluster?

-2
source

More articles:


All Articles