Hadoop HDFS Maximum File Size

Question

One of my colleagues believes that HDFS does not have a maximum file size, that is, by splitting into 128/256 megabytes any file size can be saved (it is obvious that an HDFS disk has a size, and this will be limited, but this is the only limit). I can not find anything, saying that there is a limit, is it right?

thanks jim

Answer 1

Well, obviously, the practical limit. But physically, the HDFS block identifiers are Java longs so they have a maximum of 2 ^ 63, and if your block size is 64 MB, the maximum size is 512 even.

Answer 2

I think she’s right to say that there is no maximum file size on HDFS. The only thing you can set is the block size, which is 64 MB by default. I assume that sizes of any length can be saved, the only limitation is that the larger the file size, the larger the hardware to accommodate it.

Answer 3

I am not an expert in Hadoop, but AFAIK, there is no explicit restriction on one file size, although there are implicit factors such as the total storage capacity and the maximum namespace size. In addition, there may be administrative quotes for the number of entities and directory sizes. The topic of HDFS capacity is very well described in this document . Quotes are described here and discussed here .

I would recommend paying special attention to the Michael G Noll blog, mentioned in the last link, it covers many topics about howoops.