Geek Answers Handbook

Hadoop: unable to load native-hadoop library for your platform

I installed Hadoop 2.7.2 single node in Ubuntu and I want to run the java wordcount program. Compiling and creating the jar file is successful, but when I run the jar file on Hadoop, I get this message:

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

I set the environment variables by editing the .bashrc file:

 export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib

When I type hasoop checknative -a, I get the following:

 hadoop: true /usr/local/hadoop/lib/native/libhadoop.so.1.0.0 zlib: true /lib/x86_64-linux-gnu/libz.so.1 snappy: false lz4: true revision:99 bzip2: false openssl: true /usr/lib/x86_64-linux-gnu/libcrypto.so 16/05/09 00:48:53 INFO util.ExitUtil: Exiting with status 1

Hadoop Version: 2.7.2

Ubuntu Version: 14.04

Can someone give some clues on this issue?

+5
source share
5 answers

Move your compiled source library files to the $ HADOOP_HOME / lib folder.

Then set the environment variables by editing the .bashrc file

 export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$HADOOP_HOME/lib"

Make sure your compiled source library files are in the $ HADOOP_HOME / lib folder.

He must fix the problem.

+4
source

Add command lines below hadoop-env.sh , they should suppress detected errors

 export HADOOP_HOME_WARN_SUPPRESS=1 export HADOOP_ROOT_LOGGER="WARN,DRFA"
+1
source

Try loading the chaos houses “hadoop-common.jar and hadoop-core.jar” into your class path. You can simply do this in eclipse and when you create the jar file, these bans will be automatically referenced.

0
source

This is just a warning because it cannot find the correct .jar. Either by compiling it, or because it does not exist.

If I were you, I’d just lower it

To do this, add to the appropriate configuration file

 log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR
0
source

edit the hadoop-env.sh file in /usr/local/etc/hadoop

Adding the Hadoop library to LD_LIBRARY_PATH :

 export LD_LIBRARY_PATH=/usr/local/hadoop/lib/native/:$LD_LIBRARY_PATH

try it works for me

-2
source

More articles:


All Articles