Geek Answers Handbook

The NTFS directory contains 100 thousand records. How much productivity increase if it extends to more than 100 subdirectories?

Context We have a built-in caching library with a file system. We currently have performance issues due to the large number of records (e.g. up to 100,000). Problem: we save all fs entries in one "cache directory". Very large directories work poorly.

We are considering distributing these entries into subdirectories - as git does, for example. 100 subdirectories with ~ 1000 entries each.

Question

I understand that smaller directories will help with file system access.

But it will “propagate to subdirectories” speed up the movement of all entries, for example. listing / reading of all 100,000 records? That is, when we initialize / heat the cache from the FS storage, we need to move all 100,000 entries (and delete old entries), it may take 10 + minutes.

Will "data dissemination" reduce this "transit time." In addition, this “workaround” can actually / delete obsolete entries (for example, older than N days). Will Data Dissemination Improve Deletion Time?

Optional Context - NTFS - Windows Family OS (Server 2003, 2008)

-Java J2ee application.

We / would appreciate any schooling on file system scalability.

Thanks in advance.

will be

p.s. , , , .

+5
4

, .

, : AAAA ZZZZ ( 26 ^ 4, 450K) NTFS. AA ZZ (.. ). - . .

, . , , NTFS, .

, ( ) 3 400K. , .

: NTFS, , , . , , , , , . , ( 100K) , , . , .

:. , ( AA OO). , ( ). . , , NTFS , .

+5

, ( , 100 . ), .

, , "abcdefgh.png" , "" /a/b/cdefgh.png ", " cache/abcdefgh.png ". , .

, , , , / , , "cache/2010/12/04/22/abcdefgh.png" , , AND (, ), .

+3

? Java, - Java - ( , , isn 't ..) ( ). , FindFirstFile, (, ). FindFirstFile . Java File.listFiles() . , - - . , .

- . , NTFS , n- ( ..). FAT32 , NTFS . , , . 10 15 , , , .

, ( , ) - , . (, , , , , isDirectory() - , , /file 30% ).

0

Something to see how your disk subsystem works. While drives grow rapidly, they do not get much faster (during access). This is a different disk layout (using more drives) or using SSDs. For example, an SSD has no moving parts and can touch 100K files in 10 seconds. No need to do a workout.

0
source

More articles:


All Articles