Extremely large xml file for mysql

Question

I have a 17 gb file. I want to save it in MySQL. I tried it with xmlparser in php, but it says that the maximum runtime of 30 seconds is exceeded and it only inserts a few lines. I even tried using python using the element tree, but it takes up a lot of memory, which gives a memory error in a laptop with 2 GB of RAM. Please suggest an effective way to do this.

Answer 1

You need to use Python xml.sax or lxml.etree iterparse() .

These are "event driven" xml parsing methods. You tell the parser which “node” you want to listen to, and it runs the function every time it finds that node.

It will be very low for memory usage and avoid the errors you get.

Answer 2

I would say disable runtime restriction in PHP (e.g. use CLI script) and be patient. If you say that it starts to insert something into the database from a 17 gigabyte file, it really does a good job already. There is no reason to speed it up for such a one-time job. (Increase the memory limit, just in case. By default, 128 MB is not so much.)