Do I want an ORM?

Question

We have an object model used in three applications. Two programs collect data, others read it and generate reports. The system is very disconnected, so we cannot have a single database with which all programs talk.

Currently, programs simply use a shared library to populate the object model and serialize / deserialize to disk. In particular, we use XML serialization.

There are a couple of problems with this model. 1) XML can be considered wasteful. Files can become large and bulky. Honestly, file size is not a big concern right now. 2) My biggest problem is printing a stack of memory. The entire file is loaded into the object model, managed, and then saved.

I hope that I talked about my concern, at some point we will run into memory problems with this application at runtime. Sufficient data will be collected into a single "database" (xml file), which it cannot immediately load into memory.

I would like to have access to my object model supported by file memory, not memory. I want the object model changes to be minimal. When an object accesses it, it comes from the disk, and when it is installed, it is saved (automatically, if possible).

We covered NHibernate with SQLite, SQL Compact 4.0, and EF 4 and LINQ to XML (briefly). I also used db4o in the past to cache objects to disk, but it was an unrelated project.

Before diving in and devoting time to studying one of them, I would like to know if my idea makes sense. Can I have an object model that is "magically" cached to a storage medium, and not just inflate my memory size indefinitely? What is the shortest way for this, even if it is not the most elegant?

Are there any other technologies that could help me? Files with memory mapping, linq-to-sql, Lazy (T) (only to get objects from files, when possible).

I understand that this is an open question. I am looking for a great answer to the image and details if someone there has real world experience. Links would be helpful ...

Thanks.

Answer 1

I just finished porting a (mostly "legacy") web application supported by XML files to NHibernate from exactly the same problems. I was also in the same situation that I had not used NHibernate before and did not want to learn in this process. The idea makes sense. You really can load into memory only those parts that you really need (unlike the entire database) and many other advantages.

Based on the other things you request (minimal changes to the object model that are easily portable from real applications to ORM), I'm not sure if you get them so easily. With ORMs such as NHibernate and EF4, model classes are very lightweight: they are basically a bit larger than property containers. Applications based on XML files tend to have more logic directly in the model: this logic you will probably have to move to the data access layer. Reorganizing the model and level of data access is likely to be the most time-consuming task you will have to face. I know what it was for me.

Another thing that I am taking out of your question (you say you cannot have all three programs talking to the same database and you mention SQLite and SQL Compact) is that you copy your data among your three applications by copying the file physically. How do you detect changes and how to align, do you need 3 databases? How are you currently merging changes (in case 2 out of 3 applications that you have can write data)? Depending on how you replicate the data, this ORM may or may not help you.

Change a few more points based on your comments

• If you want to combine files into one database at one moment, and you are not sure yet that this will be a Microsoft product, then NHibernate is the best choice. But if you now use LINQ extensively (according to your other comment) and want a smoother transition, I would go for EF4: Nhibernate.Linq is actually not complete, and some constructs do not work.
• Both NHibernate and EF4 are very documented and come with very good tutorials on how to create a complete application from scratch, so the tutorial is very simple.
• Both options have a “model one”, where you get a tool that creates your database for you based on your model classes, so if your model classes are very light, you can use them with a few changes.
• something that took me some time (and sometimes it’s hard for me to work), NHibernate sets up a cascade to behave as I expected: for example. What happens to "linked" objects when an object is deleted?

Answer 2

Yes, ORM maps an object model to a relational model. They do a pretty good job of hiding all the plumbing work that goes into reading and writing data to the database, caching data, managing memory, etc. They are also very capable of caching portions of object graphs and can do something to improve performance such as lazy loading data.

Answer 3

My suggestion is that you use a document database. RavenDB will give you many benefits. You could store objects without converting them to a relational model, like db4o.

However, with RavenDB you have a very rich ability to query data using Map / Reduce. Another advantage is that it is written using .NET for .NET, so you will be interested in a query in Linq. It can work as a Windows service or IIS. It can also work inline, which is great for testing. This may be a good idea for your data collection applications.

RavenDB also supports replication, so you can store data in one instance and copy it to another. Perhaps this may solve your distributed setup.

However, depending on the nature of your distributed setup, I think you might be better off using a service bus. Do you need state in data collection applications? If not, just post a message containing data on the service bus for use by another part of the system. It may seem complicated, but in fact it is not. Take a look at nServicebus .