Problem with -libjars in hadoop

Question

I'm trying to run MapReduce on Hadoop, but I got an error and I'm not sure what is going wrong. I have pas library bans that my cartographer needs.

I exit the terminal as follows:

hadoop @ubuntu: / usr / local / hadoop $ bin / hadoop jar / home / hadoop / vardtst.jar -libjars / homeop / clui.jar -libjars / homeop / model.jar gutenberg ou101

and I get the following exception:

at java.net.URLClassLoader $ 1.run (URLClassLoader.java:202)

in java.security.AccessController.doPrivileged (native method)

at java.net.URLClassLoader.findClass (URLClassLoader.java:190)

in java.lang.ClassLoader.loadClass (ClassLoader.java:306)

in java.lang.ClassLoader.loadClass (ClassLoader.java:247)

in java.lang.Class.forName0 (native method)

in java.lang.Class.forName (Class.java:247)

at org.apache.hadoop.util.RunJar.main (RunJar.java:149)

Please help .. Thanks

Answer 1

I found the answer, this caused an error due to which I was not in the main class name in the command.

The correct way to do it: hadoop @ubuntu: / usr / local / hadoop $ bin / hadoop jar / home / hadoop / vardtst.jar VardTest -libjars / home / hadoop / clui.jar, / home / hadoop / model.jar gutenberg ou101

where VardTest is the class containing the main () method.

thanks

Answer 2

It is also worth noting a subtle, but important point: the way to specify additional JARs for JVMs working with distributed cards reduces tasks, and for a JVM client it works very differently.

• -libjars makes Jars available only to JVMs that work with the remote card and reduce the task.

• For these JAR servers to be available to the JVM client (the JVM created when the jado jar command was run), you must set the HADOOP_CLASSPATH environment variable:

$ export LIBJARS=/path/jar1,/path/jar2 $ export HADOOP_CLASSPATH=/path/jar1:/path/jar2 $ hadoop jar my-example.jar com.example.MyTool -libjars ${LIBJARS} -mytoolopt value 

See: http://grepalex.com/2013/02/25/hadoop-libjars/

Another reason for -libjars to behave improperly may be the improper implementation and initialization of the custom Job class.

• The work class must implement the tool interface
• An instance of the configuration class should be obtained by calling getConf () instead of creating a new instance;

See: http://kickstarthadoop.blogspot.ca/2012/05/libjars-not-working-in-custom-mapreduce.html

Answer 3

When you specify -LIBJARS with the Hadoop jar command. First, make sure you edit your driver class as shown below:

  public class myDriverClass extends Configured implements Tool { public static void main(String[] args) throws Exception { int res = ToolRunner.run(new Configuration(), new myDriverClass(), args); System.exit(res); } public int run(String[] args) throws Exception { // Configuration processed by ToolRunner Configuration conf = getConf(); Job job = new Job(conf, "My Job"); ... ... return job.waitForCompletion(true) ? 0 : 1; } } 

Now edit the hadoop jar command as shown below:

hasoop jar YourApplication.jar [myDriverClass] args -libjars path / to / jar / file

Now let's understand what happens below. We mainly process new command line arguments by implementing the TOOL interface . ToolRunner is used to run classes that implement the Tool interface. It works in conjunction with the GenericOptionsParser to parse the hadoop command line arguments and modify the configuration of the tool.

In our Main (), we call ToolRunner.run (new configuration (), new myDriverClass (), args) - this launches this tool Tool.run (String []), after parsing with the help of general arguments . It uses the given configuration or builds one if it is null, and then sets up the configuration of the tool with a possibly modified version of conf.

Now in the run method, when we call getConf (), we get a modified version of the Configuration. So make sure you have a line at the bottom of your code. If you implement the rest and still use the conf = new Configuration () configuration, nothing will work.

Configuration conf = getConf ();