I am trying to create a site for downloading files as a side project, and I have never built anything that is needed to process a large number of files like this. As far as I can tell, there are three main options for storing and extracting files (note that there can be several files for downloading, so, for example, you can upload one or more files on website.com/a23Fc, depending on how much was originally uploaded by the user - similar to imgur.com):
- Attach all the files in one huge file directory and use the (relational) DB to determine which files belong to which URLs, and then return the list of file names depending on this. Example: a user loads website.com/abcde, so he requests a database for all files related to downloading abcde, returns their file names, and the site displays them.
- Use CouchDB because it allows you to actually attach files to individual records in the database, so each URL / upload can be a DB record with files attached to it. For example, a user downloads website.com/abcde, CouchDB captures a document with the identifier abcde, captures the files attached to this document, and transfers them to the user.
- Completely abandon the use of the database and for each download create a new directory and insert files into it. Example: a user loads site.com/abcde, a site looks for the directory / files / abcde /, captures all the files and sends them to the user, so the database is not involved at all.
Which one seems the most scalable? As I said, I have very little experience in this area, so if I am completely turned off or there is an obvious 4th option, I am more than open to it. Having thousands or millions of files in one directory (e.g. option 1) doesn't seem very smart, but having thousands or millions of directories in a directory (i.e. Option 3) doesn't look much better.