Geek Answers Handbook

Efficient XML Parsing Method in ElementTree (1.3.0) Python

I am trying to parse a huge XML file in the range of (20 MB-3 GB). Files are samples coming from different tools. So what I am doing is finding the necessary information about the item from the file and pasting them into the database (Django).

A small part of my file. A namespace exists in all files. An interesting feature of the files are more node attributes and then the text

<?xml VERSION="1.0" encoding="ISO-8859-1"?>
<mzML xmlns="http://psi.hupo.org/ms/mzml" xmlns:xs="http://www.w3.org/2001/XMLSchema-instance" xs:schemaLocation="http://psi.hupo.org/ms/mzml http://psidev.info/files/ms/mzML/xsd/mzML1.1.0.xsd" accession="plgs_example" version="1.1.0" id="urn:lsid:proteios.org:mzml.plgs_example">

    <instrumentConfiguration id="QTOF">
                    <cvParam cvRef="MS" accession="MS:1000189" name="Q-Tof ultima"/>
                    <componentList count="4">
                            <source order="1">
                                    <cvParam cvRef="MS" accession="MS:1000398" name="nanoelectrospray"/>
                            </source>
                            <analyzer order="2">
                                    <cvParam cvRef="MS" accession="MS:1000081" name="quadrupole"/>
                            </analyzer>
                            <analyzer order="3">
                                    <cvParam cvRef="MS" accession="MS:1000084" name="time-of-flight"/>
                            </analyzer>
                            <detector order="4">
                                    <cvParam cvRef="MS" accession="MS:1000114" name="microchannel plate detector"/>
                            </detector>
                    </componentList>
     </instrumentConfiguration>

Small but complete file here

So I still used findall for every element of interest.

import xml.etree.ElementTree as ET
tree=ET.parse('plgs_example.mzML')
root=tree.getroot()
NS="{http://psi.hupo.org/ms/mzml}"
s=tree.findall('.//{http://psi.hupo.org/ms/mzml}instrumentConfiguration')
for ins in range(len(s)):
    insattrib=s[ins].attrib
    # It will print out all the id attribute of instrument
    print insattrib["id"]

How can I access all the child / grandchildren of an InstruConfiguration (s) element?

s=tree.findall('.//{http://psi.hupo.org/ms/mzml}instrumentConfiguration')

Example of what i want

InstrumentConfiguration
-----------------------
Id:QTOF
Parameter1: T-Tof ultima
source:nanoelectrospray
analyzer: quadrupole
analyzer: time-of-flight
detector: microchannel plate decector

// ? find/findall ? , .

!

Edit

, !

+5
5

script, <instrumentConfiguration/> (967MB ) 40 ( ), .

24MB/s. cElementTree page (2005) 47MB/s.

#!/usr/bin/env python
from itertools import imap, islice, izip
from operator  import itemgetter
from xml.etree import cElementTree as etree

def parsexml(filename):
    it = imap(itemgetter(1),
              iter(etree.iterparse(filename, events=('start',))))
    root = next(it) # get root element
    for elem in it:
        if elem.tag == '{http://psi.hupo.org/ms/mzml}instrumentConfiguration':
            values = [('Id', elem.get('id')),
                      ('Parameter1', next(it).get('name'))] # cvParam
            componentList_count = int(next(it).get('count'))
            for parent, child in islice(izip(it, it), componentList_count):
                key = parent.tag.partition('}')[2]
                value = child.get('name')
                assert child.tag.endswith('cvParam')
                values.append((key, value))
            yield values
            root.clear() # preserve memory

def print_values(it):
    for line in (': '.join(val) for conf in it for val in conf):
        print(line)

print_values(parsexml(filename))


$ /usr/bin/time python parse_mxml.py
Id: QTOF
Parameter1: Q-Tof ultima
source: nanoelectrospray
analyzer: quadrupole
analyzer: time-of-flight
detector: microchannel plate detector
38.51user 1.16system 0:40.09elapsed 98%CPU (0avgtext+0avgdata 23360maxresident)k
1984784inputs+0outputs (2major+1634minor)pagefaults 0swaps

. , , <instrumentConfiguration/> <cvParam/> <componentList/>, .

.

ElementTree 1.3 ~ 6 , cElementTree 1.0.6.

root.clear() elem.clear(), ~ 10% , 10 . lxml.etree elem.clear(), , cElementTree, 20 (root.clear())/2 (elem.clear()) (500 ).

+4

, pymzML, python mzML. -: http://pymzml.github.com/

+3

findall, toolList. , toolList , .

0

, iterparse(). elementtree, " ".

0

, , XML, XML, .

J.F. , .

, , , .text( XML, ) ( None), . "",

it = imap(itemgetter(1),
          iter(etree.iterparse(filename, events=('end',))))
root = next(it) # get root element

- xml ( ), , "end", "".

, , . . .

XML :

<data>
<country>
    <name>Liechtenstein</name>
    <rank>1</rank>
    <year>2008</year>
    <gdppc>141100</gdppc>
</country>
<country>
    <name>Singapore</name>
    <rank>4</rank>
    <year>2011</year>
    <gdppc>59900</gdppc>
</country>
<country>
    <name>Panama</name>
    <rank>68</rank>
    <year>2011</year>
    <gdppc>13600</gdppc>
</country>

0

More articles:


All Articles