How can I find a large XML dataset?

Question

I have a DataModule with XML and I need a search ...

Unfortunately, there are over 300,000 entries, and I cannot do a loop to check one by one.

Can I make a request without using a database?

Is there any other solution?

Answer 1

XML is great for small amounts of information, but for a large dataset, a relational database is really the only smart choice, especially if you need to be able to query it.

Answer 2

You can search using something like XPath: however, this would mean that the XPath implementation is searching on your behalf (which does not necessarily improve performance).

Answer 3

I think it's probably important to ask why you use XML to store 300,000 records. Since XML is not the most efficient format for managing data using.

If you are stuck in XML, it might be best for you to read the XML file into some kind of database (you can leave with a memory table, but then you might run out of memory). I think that if you use the TXMLDocument object to load an XML file, you will have a serious performance problem or run out of memory (I had problems when I played with the xml x 250k file some time ago).

Perhaps you can directly use the MSXML DOM (perhaps you can import the type library) or use SAX, which will allow you to parse it sequentially, none of which I have had much experience.

Answer 4

There are several in-memory databases that may be useful. At the very least, you could then index and query the data as needed. I know one of them from the components of 4developers.com. David

Answer 5

You do not say how you implement the data source. I used a TClientDataSet connected via TXMLTransformProvider (OK not for 300K records), but for several thousand. and just setting filters and filtered properties seems to be "Query", it's just great ...

Or am I missing something?

Answer 6

For SAX parsing for Delphi, set this question in Stackoverflow:

Is there a SAX Parser for Delphi and Free Pascal?