Large XML files and pagination, is this possible?

Question

Problem

When opening very large XML files locally, on your machine, it is almost certain that it will take an age to open this file - this can often mean that your computer is locked because it thinks it is not responding.

This is a problem if you serve users with XML backups of rather complex databases or systems that they use - the similarity with them, which can open large backups, not to mention their use, is subtle.

Is pagination possible?

I use XSLT to present readable backups to users. In the same way, only a page could be pulled out during the data to prevent the entire file from being read at a time, thereby causing problems above.

I assume that the answer is simply no, but I would like to know if anyone saw other problems and resolved them.

Note It is only on the local machine, it does not require an Internet connection. JavaScript can be used if this makes things easier.

Answer 1

It is possible to paginate using XSLT, but probably will not produce the desired results: for XSLT to work, the entire XML document must be parsed into the DOM tree.

What you can do is experiment with stream conversions: http://stx.sourceforge.net/

Or you can pre-process a large XML file to cut it into smaller bits before processing with XSLT. For this I would use a command line tool like XMLStarlet

Answer 2

Go ahead, a very good question!

XSLT implementations that I know require a DOM, so they need to access the entire document (although it could be done in a lazy way)

In any case, you should take a look at VTD-XML: http://vtd-xml.sourceforge.net/

The latest SAXON XSLT processor also supports rudimentary support for the so-called "XSLT streaming." Read about it here: http://www.saxonica.com/documentation/index/intro.html

However, database backups are probably not suitable for use in XML. If you need to deal with XML database backups, I will try to get away from them as quickly as possible. The same goes for magazines - a linear process should work just by adding things. I mean, it would be even better if XML allows the forest as a top-level structure, but I think this will never happen.

Answer 3

XMLMax A virtual xml editor will read, parse and display a 1 gigabyte XML file in a tree structure in about 30 seconds on a fast PC. Windows only. It will work with xml of any size or structure.

Answer 4

HI, I don’t know which programming language you are using, but in C # using XMLReader I can read the file tag by tag, not the whole file. This way you can only read the first page and stop reading. Best regards, Jordan

Answer 5

One way to alleviate this problem would be to split the large XML files into several smaller XML documents. Depending on the type of data, you can split or split the file into any number of ways (for example, day, transaction, entity, etc.).

This, of course, will cause a number of other problems. For example, you will need to create a specialized parser if you need to view the data as a whole or through sections.