How to provide a huge number of files as part of your application

Question

So, my application depends on a large number of small files. The actual number is around 90,000. Now I use a component that needs access to these files, but the only way it accepts them is to use a URI.

So far, I just added a directory containing all the files to the debug folder when I developed the application. However, I must now consider deployment. What are my options for including all of these files in my deployment?

So far, I have come up with a couple of different solutions, none of which I have been able to fully implement. At first it was easy to add all the files to the installer, which would then copy them to their places. It is theoretically, at least, work, but it would make the installation of the installer (the standard MSI installer generated with VS) an absolute hell.

The next option I came across was to pin them into a single file and add this as part of the installer, and then unzip them using a special action. The standard libraries do not seem to support complex zip files, which makes this option quite complicated.

Finally, I realized that I could create a separate project and add all the files as resources in this project. I do not know how URIs work that point to resources stored in other assemblies. Sense, is it "standard" for everything that supports the "application: // ,, Assembly" format?

So, are these the only options I have, or are there some others? And what would be the best option for this?

Answer 1

I would use one ZIP archive file, and not unzip this file on your hard drive, but leave it as it is. This is also the approach used by several well-known applications that depend on a large number of small files.

Windows supports the use of zip files as virtual folders (with XP), users can see and edit their contents with standard tools such as Windows Explorer.

C # also has excellent support for zip files, if you are not comfortable with the built-in tool, I recommend one of the main Zip libraries - they are very easy to use.

If you're concerned about performance, caching files in memory is a simple exercise. If your use case actually requires the presence of files on the disk, also not a problem, just unzip them at first use - these are just a few lines of code.

In short, just use the zip archive and a good library, and you will not run into any problem.

In any case, I would not insert this huge number of files into your application directly. Data files must be separate.

Answer 2

You can include the files in the zip archive and attach them to the first run yourself as part of the final configuration, if it is not practical to do this from the installer. This is not entirely atypical (for example, it seems that most Microsoft applications install the configuration after installation the first time they start).

Depending on how resources are used, you might have a service that provides them on demand from some storage and caches them, rather than dumping them somewhere. This may or may not make sense, depending on what these resources are intended for, for example, if they are user interface elements, the delay at first access may not be acceptable.

You can even serve them using http from a local or non-local server or SQL server if it already uses one, caching them, which would be useful for maintenance, but might not work for the environment.

I would not do anything that included an embedded resource for each file individually, and that would be hell for support.

Answer 3

Another option would be to create a zip / rar archive with a self-extracting archive and extract it from the installer.

Answer 4

One option is to save them in a complex repository and access them directly in the repository. An article on our site describes the different types of storage and their advantages / features.