Hadoop: error setting object

Question

I try to run Terasort testers and I get the following exception:

java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.MapTask$OldOutputCollector.<init>(MapTask.java:573) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371) at org.apache.hadoop.mapred.Child$4.run(Child.java:259) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88) ... 10 more Caused by: java.lang.IllegalArgumentException: can't read paritions file at org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.configure(TeraSort.java:213) ... 15 more Caused by: java.io.FileNotFoundException: File _partition.lst does not exist. at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245) at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:720) at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417) at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412) at org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.readPartitions(TeraSort.java:153) at org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.configure(TeraSort.java:210) ... 15 more 

TeraGen commands work fine and created input files for TeraSort. Here is a list of my input directory:

 bin/hadoop fs -ls /user/hadoop/terasort-input/Warning: Maximum heap size rounded up to 1024 MB Found 5 items -rw-r--r-- 1 sqatest supergroup 0 2012-01-23 14:13 /user/hadoop/terasort-input/_SUCCESS drwxr-xr-x - sqatest supergroup 0 2012-01-23 13:30 /user/hadoop/terasort-input/_logs -rw-r--r-- 1 sqatest supergroup 129 2012-01-23 15:49 /user/hadoop/terasort-input/_partition.lst -rw-r--r-- 1 sqatest supergroup 50000000000 2012-01-23 13:30 /user/hadoop/terasort-input/part-00000 -rw-r--r-- 1 sqatest supergroup 50000000000 2012-01-23 13:30 /user/hadoop/terasort-input/part-00001 

Here is my command to run terasort:

 bin/hadoop jar hadoop-examples-0.20.203.0.jar terasort -libjars hadoop-examples-0.20.203.0.jar /user/hadoop/terasort-input /user/hadoop/terasort-output 

I see the _partition.lst file in my input directory, I do not understand why I get a FileNotFoundException.

I followed the settings presented at: http://www.michael-noll.com/blog/2011/04/09/benchmarking-and-stress-testing-an-hadoop-cluster-with-terasort-testdfsio-nnbench -mrbench /

Answer 1

The problem arose because I deployed the job to NFS. I changed hasoop.tmp.dir to point it to the local file system (/ tmp) and the problem went away in jiffy.

Answer 2

I got this to work as follows:

I work in local mode from my hadoop base directory, hasoop-1.0.0 with an input subdirectory below it, and I get the same error as you.

I edited a damaged java file to make it write the path instead of the file name, rebuild it ("ant binary") and re-run it. He was looking for a file in the directory in which I was working. I have no idea if he looked in the hadoop base directory or in the executable directory.

... so I made a symbolic link in the directory in which I ran terasort, pointing to the real file in the input directory.

This is a cheap hack, but it works.

 - Tim. 

Answer 3

Do you have a setting to run in pseudo-distributed mode (or in a real cluster)? If you do not configure Hadoop, it will work in local mode (like libs inside the same process). Terasort does NOT work in LocalJobRunner mode. Look for the word LocalJobRunner for output.

Here is a link to configure HDFS, SSH and rsync: http://hadoop.apache.org/docs/r1.1.1/single_node_setup.html#PseudoDistributed

Answer 4

I am using cloudera CDH4. ran into a similar problem with my other hadoop program. I believe that problems relate to linking external libraries.

The program worked fine in Eclipse (local mode), but when I tried to run it in pseudo-distributed mode, I received this error message.

Workaround: - Created a jar file from Eclipse with the ability to process the library - copy the necessary libraries to a subfolder next to the created JAR. - Copied the JAR file to the hadoop home directory (the path where the hadoop-exampls.jar files are located)

with this fix, I can run the hadoop program without any errors. hope this helps