Geek Answers Handbook

Copying a directory from the local system to hdfs java code

I had a problem copying a directory from the local system to HDFS using java code. I can move individual files, but I cannot find a way to move the entire directory with subfolders and files. Can someone help me with this? Thanks in advance.

+8
java hadoop hdfs
source share
2 answers

Just use the FileSystem copyFromLocalFile method. If the source path is a local directory, it will be copied to the HDFS destination:

 ... Configuration conf = new Configuration(); conf.addResource(new Path("/home/user/hadoop/conf/core-site.xml")); conf.addResource(new Path("/home/user/hadoop/conf/hdfs-site.xml")); FileSystem fs = FileSystem.get(conf); fs.copyFromLocalFile(new Path("/home/user/directory/"), new Path("/user/hadoop/dir")); ...
+17
source share

Here is the full working code for reading and writing to HDFS. It takes two arguments

  • Input Path (Local / HDFS)

  • Output Path (HDFS)

I used the Cloudera sandbox.

  package hdfsread; import java.io.BufferedInputStream; import java.io.FileInputStream; import java.io.IOException; import java.io.InputStream; import java.io.OutputStream; import java.net.URI; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IOUtils; public class ReadingAFileFromHDFS { public static void main(String[] args) throws IOException { String uri = args[0]; InputStream in = null; Path pt = new Path(uri); Configuration myConf = new Configuration(); Path outputPath = new Path(args[1]); myConf.set("fs.defaultFS","hdfs://quickstart.cloudera:8020"); FileSystem fSystem = FileSystem.get(URI.create(uri),myConf); OutputStream os = fSystem.create(outputPath); try{ InputStream is = new BufferedInputStream(new FileInputStream(uri)); IOUtils.copyBytes(is, os, 4096, false); } catch(IOException e){ e.printStackTrace(); } finally{ IOUtils.closeStream(in); } } }
0
source share

More articles:


All Articles