Moving data in hdf using copyFromLocal switch

Question

I do not know what is happening here, but I am trying to copy a simple file from a directory in my local file system to the directory specified for hdfs.

In my hdfs-site.xml, I indicated that the directory for hdfs would be / home / vaibhav / Hadoop / dataNodeHadoopData using the following properties -

<name>dfs.data.dir</name> <value>/home/vaibhav/Hadoop/dataNodeHadoopData/</value> and <name>dfs.name.dir</name> <value>/home/vaibhav/Hadoop/dataNodeHadoopData/</value> 

I use the following command -

 bin/hadoop dfs -copyFromLocal /home/vaibhav/ml-100k/u.data /home/vaibhav/Hadoop/dataNodeHadoopData 

copy the u.data file from it to the local file system folder in the directory that I specified as the Hdfs directory. But when I do this, nothing happens - there is no mistake, nothing. And the file is not copied to hdsf. Am I doing something wrong? Are any permissions possible?

Necessary suggestions.

I use pseudo-distributed single mode node.

In addition, in the corresponding note, I want to ask that in my map reduction program I set the configuration to point to inputFilePath as / home / vaibhav / ml -100k / u.data. So will it not automatically copy the file from the given location to hdf?

Answer 1

I believe that dfs.data.dir and dfs.name.dir should point to two different and existing directories. Also, make sure you format the FS naming after changing directories in the configuration.

When copying to HDFS, you incorrectly specify the target. The correct syntax for copying a local file to HDFS is:

 bin/hadoop dfs -copyFromLocal <local_FS_filename> <target_on_HDFS> 

Example:

 bin/hadoop dfs -copyFromLocal /home/vaibhav/ml-100k/u.data my.data 

This will create the my.data file in the my.data 's home directory in HDFS. Before you copy files to HDFS, make sure that you first create the contents of the directory and create the directory.