Quick access to files in a directory with 500,000 files

Question

I have a directory with 500,000 files. I would like to access them as soon as possible. The algorithm requires that I open and close them many times (cannot simultaneously open 500,000 files).

How can I do this efficiently? Initially, I thought I could cache inodes and open files this way, but * nix does not provide a way to open inodes (security or some).

Another option is to simply not worry about it and hope that FS does a good job of the file in the directory. If this is the best option, then FS will work best. Are some file name patterns faster than others? e.g. 01234.txt vs foo.txt

By the way, all this is on Linux.

Answer 1

Assuming your ext3 file system , your directory is indexed by a hashed B-tree if dir_index is enabled. This will give you as much incentive as anything you could add to your application.

If the directory is indexed, your file naming scheme does not matter.

http://lonesysadmin.net/2007/08/17/use-dir_index-for-your-new-ext3-filesystems/

Answer 2

A few ideas:

a) If you can control the layout of the directory, place the files in subdirectories.

b) , , , xfs ?

Answer 3

, ulimit , , 100 000 , 500 000 .

, , dentry . dentry - → inode, ​​ , dentry, . 2.6 256 * , 2 , 500 000 .

, , , , .

Answer 4

- . , , . 256 (, , 012345678 01/2345678). , .

, , , .

Answer 5

Another question: how much data is in the files? Is an SQL completion option an option?