Parsing large XML files in Adobe Flex

Question

I am working on an Adobe Flex application that needs to parse a large relativistic XML file. ATM is only 35 MB, but in an ideal world there will be much more in the future. ** Edit: I have no control over the XML file

I essentially drop the content directly into the SQLITE database, so I can use the SimpleXML class to turn it into an object and then iterate over it, but I'm worried that this will be a bad approach as the file gets bigger, I'm paranoid, or is there a better one way to do it?

Answer 1

You will definitely encounter some performance issues when parsing a large XML file. Back in Flex for 2 days, we used SOAP for services and had one data call that canceled about 5K records, and Flash Player freezes / the browser does not respond for 10 seconds on a fairly fast machine. I cannot remember the size of this SOAP message, but it could not exceed 1-2 MB.

If your backend can convert XML to an object graph and send it back to AMF, you will see much better performance. Flash Player does a great job with large data sets, provided they are encoded in AMF (compressed binary format).

Even, I would really think if you want to send the only result that can break it into pieces. At least you have a way to scale better and can give the user more accurate feedback, that is, display a message such as "Processing element 6 of 35 ..."

Answer 2

What you want is a SAX XML parser - it can parse a stream without reading all of this. I can not find it for AS3, though (although there are other people who are looking for the same thing).

SAX works by raising events when elements cross the input stream. Quite convenient - I often used it in the past, and as soon as you are familiar with it, it is useful for many cases. It works even with an open socket, where the stream never closes.

Answer 3

Personally, I will try to avoid XML files of this size. This is one of the drawbacks of XML, you need to read the entire file before you can use it.

Can't you get your database back in smaller pieces of XML, with what you need, and not right away? It may be a little harder to set up, but it can turn out to be much more scalable.

Answer 4

As others have said, parsing large amounts of XML is not suggested and can become rather sluggish. It has been stated that the fastest way to send data between the flash client and the server side of the script is the AMF format (action message format), which is binary. If you have ever done anything with the SharedObject class, then you already had some relationships with AMF, as this is the format that it writes LSO to your hard drive. AMFPHP was the best solution for this until recently, as it now lends itself to the Zend framework, more specifically now ZendAMF.

There is an excellent tutorial here , Lee Brimelou, one of the flash developers for whom I am looking for inspiration and clarity, shows how to use ZendAMF.

The speed with which your data is available in ZendAMF, compared to the usual old XML, is staggering, and the more the data being analyzed, the more noticeable.

Answer 5

As already mentioned, the SAX parser will be your best choice so that you can process each "event" (node) as you read it, instead of using the DOM parser to read the entire XML file and store it in memory.

But if you intend to use such large data sets, perhaps you could consider exporting SQLite data in JSON format rather than XML?

I don’t know exactly how to directly export SQLite to JSON (without writing my own script), however the message on the sqlite-users mailing list suggests trying the following unsupported / undocumented source code: http://www.ch-werner.de/sqliteodbc/sqlite3json .tgz

A tutorial on using JSON in Flex can be found at http://www.mikechambers.com/blog/2006/03/28/tutorial-using-json-with-flex-2-and-actionscript-3/

Answer 6

There is always a WHERE clause in SQL because no one ever wants to see more than 100 results.

You may not have control over the source XML file, but perhaps you can embed something on the server side that parses and retrieves the data you really want.

Greetings

Answer 7

For comparing the speed between XML and JSON, I actually made a comparison between 3 types for speed for big big data - XML, JSON and BlazeDS. Trust me, BlazeDS will be the fastest. It is really faster.