Geek Answers Handbook

Get hadoop configuration in Java

I am writing a Java utility that needs to access DFS, so I need a Configuration object. When I create it just using

Configuration conf = new Configuration()

it does not find DFS and simply uses the local file system; Print

fs.getHomeDirectory()

provides a local home directory. I tried to add core-site.xml, mapred-site.xml, yarn-site.xml and hdfs-site.xml for configuration as resources, but nothing changes. What do I need to do to get the HDFS settings?

thank you for reading

+6
source share
2 answers

The reason this points to your local file system is that core-site.xml and hdfs-site.xml not added properly. The code snippet below will help you.

 Configuration conf = new Configuration(); conf.addResource(new Path("file:///etc/hadoop/conf/core-site.xml")); // Replace with actual path conf.addResource(new Path("file:///etc/hadoop/conf/hdfs-site.xml")); // Replace with actual path Path pt = new Path("."); // HDFS Path FileSystem fs = pt.getFileSystem(conf); System.out.println("Home directory :"+fs.getHomeDirectory());

Update:

An option should work, It seems there are some problems in the configuration file or path. You have another option, instead of adding configuration files using the addResource method, use the set method. Open core-site.xml and look for fs.defaultFS . Use the set method instead of the addResource method.

 conf.set("fs.defaultFS","hdfs://<Namenode-Host>:<Port>"); // Refer you core-site.xml file and replace <Namenode-Host> and <Port> with your cluster namenode and Port (default port number should be `8020`).
+12
source

To access the File System, you must use the configuration and file system as described below.

  • Get configuration instance

  • Get an instance of HDFS

     Configuration configuration = new Configuration(); FileSystem hdfs = FileSystem.get(new URI("hdfs://"+HadoopLocation+":8020"), configuration);

In this case, HadoopLocation is the location where you had the hadoop server (possibly Localhost)

+2
source

More articles:


All Articles