Getting an error when copying a file from s3: // to the local (hasoop) file system

Question

I am trying to copy files from s3 to hadoop file system using python. I got the following error:

cp: `foo/ds=2015-02-13/ip-d1b-request-2015-02-13_10-00_10-09.txt.gz': No such file or directory 

I recently translated the latest version of howop (2.4.0). In version (0.20) it works fine. Why am I getting this error in version 2.4.0?

In Hadoop Version 0.20

 hadoop@ip-10-76-38-167 :~$ /home/hadoop/bin/hadoop fs -cp s3://test.com/foo/ds=2015-02-13/ip-d1b-request-2015-02-13_10-00_10-09.txt.gz /foo/ds=2015-02-13/ip-d1b-request-2015-02-13_10-00_10-09.txt.gz 15/02/13 11:21:45 INFO s3native.NativeS3FileSystem: Opening 's3://test.com/foo/ds=2015-02-13/ip-d1b-request-2015-02-13_10-00_10-09.txt.gz' for reading 

In version Hadoop 2.4.0

 [ hadoop@ip-10-169-19-123 ~]$ /home/hadoop/bin/hadoop fs -cp s3://test.com/foo/ds=2015-02-13/ip-d1b-request-2015-02-13_10-00_10-09.txt.gz /foo/ds=2015-02-13/ip-d1b-request-2015-02-13_10-00_10-09.txt.gz 15/02/13 11:21:37 INFO guice.EmrFSBaseModule: Consistency disabled, using com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem as FileSystem implementation. 15/02/13 11:21:38 INFO fs.EmrFileSystem: Using com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem as filesystem implementation cp: `foo/ds=2015-02-13/ip-d1b-request-2015-02-13_10-00_10-09.txt.gz': No such file or directory 

Answer 1

I found the answer myself.

 Using `distcp` instead of `fs -cp`. 

This command works without any problems.

Answer 2

You need to try like this. Add "

 /home/hadoop/bin/hadoop fs -cp "s3://test.com/foo/ds=2015-02-13/ip-d1b-request-2015-02-13_10-00_10-09.txt.gz" "/foo/ds=2015-02-13/ip-d1b-request-2015-02-13_10-00_10-09.txt.gz"