Difference between Amazon S3 and S3n in Hadoop

Question

When I connected my Hadoop cluster to the Amazon repository and uploaded the file to HDFS, I found that s3: // was not working but was looking for some help on the Internet. I found that I can use S3n, so when I used S3n it worked. I do not understand the differences between using S3 or s3n with my hadoop cluster, can anyone explain?

Answer 1

I think the main problem is having S3 and S3N two separate connection points for Hadoop. S3n: // means "A regular file read from the outside world on this S3-url." S3: // refers to the HDFS file system displayed in the S3 bucket that resides on the AWS storage cluster. Therefore, when you used the file from the Amazon repository, you should use S3N and why your problem has been resolved. Also added info added by @Steffen.

Answer 2

Two file systems for using Amazon S3 are documented in the corresponding Hadoop wiki page addressed to Amazon S3 :

• S3 Native FileSystem (URI scheme: s3n)
  Own file system for reading and writing regular files on S3. The advantage of this file system is its access to S3 files that were written using other tools. Conversely, other tools can access files written using Hadoop. The downside is the 5 GB limit on file size imposed by S3 . For this reason, it is not suitable for replacing HDFS (which has support for very large files).

• S3 Block FileSystem (URI scheme: s3)
  Block based file system supported by S3. Files are stored in blocks, just like they are in HDFS. This allows you to efficiently rename. This file system needs to allocate a bucket for the file system - you should not use an existing bucket containing files, or write other files in the same bucket. Files stored on this file system can be more than 5 GB, but they are not compatible with other S3 tools .

There are two ways to use S3 with Hadoop Map / Reduce, either as a replacement for HDFS using the S3 block file system (i.e. using it as a reliable distributed file system supporting very large files) or as a convenient repository for data input and output from MapReduce. using the S3 file system. In the second case, HDFS is still used for the Map / Reduce phase. [...]

[emphasis mine]

So the difference is mainly related to how the 5 GB limit is handled (which is the largest object that can be loaded in one PUT, although the objects can have a size of 1 to 5 terabytes, see How much data can I store? ) : when using S3 Block FileSystem (URI scheme: s3) allows you to fix the 5 GB limit and store files up to 5 TB, this replaces HDFS in turn.

Answer 3

Here is an explanation: https://notes.mindprince.in/2014/08/01/difference-between-s3-block-and-s3-native-filesystem-on-hadoop.html

Hadoop 0.10.0 (HADOOP-574) introduced the first Hadoop file system with S3 support. It was called the file system of the S3 block, and the URI s3: // scheme was assigned to it. In this implementation, files are stored as blocks, as in HDFS. Files stored in this file system are not compatible with other S3 tools - this means that if you go to the AWS console and try to find files written by this file system, you will not find them, instead you will find files with names like block_ -1212312341234512345 etc.

To overcome these limitations, another S3-enabled file system was introduced in Hadoop 0.18.0 (HADOOP-930). It was called the S3 native file system and was assigned the s3n: // URI scheme. This file system allows you to access files on S3 that were written using other tools ... When this file system was introduced, S3 had a file size limit of 5 GB and therefore this file system could only work with files smaller than 5 GB At the end of 2010, Amazon ... increased the file size from 5 GB to 5 TB ...

Using the S3 block file system is no longer recommended. Various Hadoop-as-a-service providers, such as Qubole and Amazon EMR, have come to match both s3: // and s3n: // URIs for their own S3 file system to provide this.

Therefore, always use your own file system. 5Gb limits no longer. Sometimes you may need to enter s3:// instead of s3n:// , but just make sure that all the files you created are visible in the browser browser.

Also see http://docs.aws.amazon.com/ElasticMapReduce/latest/ManagementGuide/emr-plan-file-systems.html .

Amazon EMR previously used the S3 Native FileSystem with the s3n URI. Although this still works, we recommend using the s3 URI scheme for maximum performance, security, and reliability.

It also says that you can use s3bfs:// to access the old block file system, formerly known as s3:// .