Geek Answers Handbook

The optimal number of files in a directory and the number of directories for EXT4

I have a program that produces a large number of small files (say, 10,000 files). After they are created, another script accesses them and processes them one by one.

Questions:

  • Does performance matter how the files are organized (all in one directory or in multiple directories)
  • if so, what is the optimal number of directories and files in a directory?

I am running Debian with the ext4 file system

Similar

+8
linux ext4
source share
1 answer

10k files inside one folder is not a problem on Ext4. It should have a default dir_index parameter that indexes the contents of directories using a structure of type btree to prevent performance problems.

To summarize, if you are not creating millions of files or using ext2 / ext3, you do not have to worry about system or FS performance issues.

At the same time, shell tools and commands do not like to be called with a large number of files as a parameter (for example, rm * ) and may return an error message saying something like "too many arguments". Look at this answer for what happens next.

+8
source share

More articles:


All Articles