Geek Answers Handbook

How can I use a very large dictionary in C #?

I want to use a search map or dictionary in a C # application, but it is expected that it will store 1-2 GB of data.

Can someone please tell me if I can still use the dictionary class, or if I need to use some other class?

EDIT: We have an existing application that uses the oracle database to query or search for object details. This, however, is too slow, because the same objects are repeatedly requested. I felt that it might be ideal to use a search map for this scenario to improve response time. However, I worry if size makes this a problem.

+6
c # memory
source share
6 answers

Short answer

Yes. If your computer has enough memory for the structure (and the overhead of the rest of the program and system, including the operating system).

Long answer

Do you really want? Without knowing more about your application, it is difficult to understand what to offer.

  • Where does the data come from? File? Files? Database? Services?
  • Is this a caching mechanism? If so, can you expire items from the cache once they have not been available for some time? Thus, you do not need to keep in mind all the time.
  • As others suggested, if you are just trying to store a lot of data, can you just use the database? Thus, you do not have to have all the information in memory at once. When indexing, most databases do a great job of quick retrieval. You can combine this approach with a cache.
  • Is the data that will be in read-only memory, or should it be stored in some kind of memory when something changes?
  • Scalability - Do you expect the amount of data that will be stored in this dictionary to increase over time? If so, you will find it very expensive to buy machines that can handle this amount of data. You might want to look at a distributed caching system if that is the case (AppFrabric comes to mind), so you can scale horizontally (more machines) and not vertically (one very large expensive point of failure).

UPDATE

In light of editing the poster, it looks like caching here will be long. There are many ways to do this:

  • Simple word caching - just a cache file as it is requested.
  • Memcache
  • Application block caching I'm not a big fan of this implementation, but others have been successful.
+7
source share

While you are working on a 64-bit machine, yes, you should be able to use this dictionary. However, if you have SUCH a lot of data, the database may be more suitable (cassandra is really nothing but a giant dictionary, and there is always MySQL).

+1
source share

When you say 1-2 GB of data, I assume that you mean that objects are complex objects that collectively contain 1-2 GB.

If they are not structures (and should not be), the dictionary does not care about how large the elements are.
As long as you have less than about 2 24 elements (I pulled this number from my hat), you can save as much as you can fit in memory.

However, like everyone else, you should probably use a database. You might want to use an in-memory database such as SQL CE.

+1
source share

You can, but for a dictionary, how much better is it, use DataBase

0
source share

Use the database. Make sure you have a good database model, put the correct indexes and release.

0
source share

You can use subdictionaries .

 Dictionary<KeyA, Dictionary<KeyB ....

Where KeyA is some common part of KeyB .

For example, if you have a String dictionary, you can use the first letter as KeyA .

-2
source share

More articles:


All Articles