All geek questions in one place

Xml analysis in python using ElementTree

I am very new to python and I need to parse some dirty XML files that need to be sanitized first.

I have the following python code:

import arff import xml.etree.ElementTree import re totstring="" with open('input.sgm', 'r') as inF: for line in inF: string=re.sub("[^0-9a-zA-Z<>/\s=!-\"\"]+","", line) totstring+=string data=xml.etree.ElementTree.fromstring(totstring) print data file.close

which analyzes:

 <!DOCTYPE lewis SYSTEM "lewis.dtd"> <REUTERS TOPICS="YES" LEWISSPLIT="TRAIN" CGISPLIT="TRAINING-SET" OLDID="5544" NEWID="1"> <DATE>26-FEB-1987 15:01:01.79</DATE> <TOPICS><D>cocoa</D></TOPICS> <PLACES><D>el-salvador</D><D>usa</D><D>uruguay</D></PLACES> <PEOPLE></PEOPLE> <ORGS></ORGS> <EXCHANGES></EXCHANGES> <COMPANIES></COMPANIES> <UNKNOWN> &#5;&#5;&#5;CT &#22;&#22;&#1;f0704&#31;reute uf BC-BAHIA-COCOA-REVIEW 02-26 0105</UNKNOWN> <TEXT>&#2; <TITLE>BAHIA COCOA REVIEW</TITLE> <DATELINE> SALVADOR, Feb 26 - </DATELINE><BODY>Showers continued throughout the week in the Bahia cocoa zone, alleviating the drought since early January and improving prospects for the coming temporao, although normal humidity levels have not been restored, Comissaria Smith said in its weekly review. The dry period means the temporao will be late this year. Arrivals for the week ended February 22 were 155,221 bags of 60 kilos making a cumulative total for the season of 5.93 mln against 5.81 at the same stage last year. Again it seems that cocoa delivered earlier on consignment was included in the arrivals figures. Comissaria Smith said there is still some doubt as to how much old crop cocoa is still available as harvesting has practically come to an end. With total Bahia crop estimates around 6.4 mln bags and sales standing at almost 6.2 mln there are a few hundred thousand bags still in the hands of farmers, middlemen, exporters and processors. There are doubts as to how much of this cocoa would be fit for export as shippers are now experiencing dificulties in obtaining +Bahia superior+ certificates. In view of the lower quality over recent weeks farmers have sold a good part of their cocoa held on consignment. Comissaria Smith said spot bean prices rose to 340 to 350 cruzados per arroba of 15 kilos. Bean shippers were reluctant to offer nearby shipment and only limited sales were booked for March shipment at 1,750 to 1,780 dlrs per tonne to ports to be named. New crop sales were also light and all to open ports with June/July going at 1,850 and 1,880 dlrs and at 35 and 45 dlrs under New York july, Aug/Sept at 1,870, 1,875 and 1,880 dlrs per tonne FOB. Routine sales of butter were made. March/April sold at 4,340, 4,345 and 4,350 dlrs. April/May butter went at 2.27 times New York May, June/July at 4,400 and 4,415 dlrs, Aug/Sept at 4,351 to 4,450 dlrs and at 2.27 and 2.28 times New York Sept and Oct/Dec at 4,480 dlrs and 2.27 times New York Dec, Comissaria Smith said. Destinations were the US, Covertible currency areas, Uruguay and open ports. Cake sales were registered at 785 to 995 dlrs for March/April, 785 dlrs for May, 753 dlrs for Aug and 0.39 times New York Dec for Oct/Dec. Buyers were the US, Argentina, Uruguay and convertible currency areas. Liquor sales were limited with March/April selling at 2,325 and 2,380 dlrs, June/July at 2,375 dlrs and at 1.25 times New York July, Aug/Sept at 2,400 dlrs and at 1.25 times New York Sept and Oct/Dec at 1.25 times New York Dec, Comissaria Smith said. Total Bahia sales are currently estimated at 6.13 mln bags against the 1986/87 crop and 1.06 mln bags against the 1987/88 crop. Final figures for the period to February 28 are expected to be published by the Brazilian Cocoa Trade Commission after carnival which ends midday on February 27. Reuter &#3;</BODY></TEXT> </REUTERS>

How now can I get only text from the body tag?

All the tutorials I've seen rely on reading xml directly from a file, so Elementtree.parse works. Since I'm trying to parse from a string, this will not work, and this breaks a lot of the tutorials I read.

Many thanks

+4
source share
3 answers

If you don't care about the specific structure of the (potentially messy) XML document and just want to quickly get the contents of this tag / element, you can try BeautifulSoup .

 import BeautifulSoup from BeautifulSoup import BeautifulSoup soup = BeautifulSoup(totstring) body = soup.find("body") bodytext = body.text
+5
source

Your first clue may be when you receive such messages ...

 >>> from xml.etree import ElementTree >>> parse = ElementTree.parse('foo.xml') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python2.6/xml/etree/ElementTree.py", line 862, in parse tree.parse(source, parser) File "/usr/lib/python2.6/xml/etree/ElementTree.py", line 586, in parse parser.feed(data) File "/usr/lib/python2.6/xml/etree/ElementTree.py", line 1245, in feed self._parser.Parse(data, 0) xml.parsers.expat.ExpatError: reference to invalid character number: line 11, column 0 >>>

This error comes from invalid characters in the XML source. You need to clear invalid characters (see fix_xml.py at the bottom of my answer).

Once you have pure XML, it's pretty easy. You should use StringIO to treat strings as files:

 >>> from xml.etree import ElementTree >>> from StringIO import StringIO >>> text = open('foo.xml', 'r').read() >>> tree = ElementTree.parse(StringIO(text)) >>> tree.find('//BODY') <Element BODY at b723cf2c> >>> tree.find('//BODY').text 'Showers continued throughout the week in\nthe Bahia cocoa zone, alleviating the drought since early\nJanuary and improving prospects for the coming temporao,\nalthough normal humidity levels have not been restored,\nComissaria Smith said in its weekly review.\n The dry period means the temporao will be late this year.\n Arrivals for the week ended February 22 were 155,221 bags\nof 60 kilos making a cumulative total for the season of 5.93\nmln against 5.81 at the same stage last year. Again it seems\nthat cocoa delivered earlier on consignment was included in the\narrivals figures.\n Comissaria Smith said there is still some doubt as to how\nmuch old crop cocoa is still available as harvesting has\npractically come to an end. With total Bahia crop estimates\naround 6.4 mln bags and sales standing at almost 6.2 mln there\nare a few hundred thousand bags still in the hands of farmers,\nmiddlemen, exporters and processors.\n There are doubts as to how much of this cocoa would be fit\nfor export as shippers are now experiencing dificulties in\nobtaining +Bahia superior+ certificates.\n In view of the lower quality over recent weeks farmers have\nsold a good part of their cocoa held on consignment.\n Comissaria Smith said spot bean prices rose to 340 to 350\ncruzados per arroba of 15 kilos.\n Bean shippers were reluctant to offer nearby shipment and\nonly limited sales were booked for March shipment at 1,750 to\n1,780 dlrs per tonne to ports to be named.\n New crop sales were also light and all to open ports with\nJune/July going at 1,850 and 1,880 dlrs and at 35 and 45 dlrs\nunder New York july, Aug/Sept at 1,870, 1,875 and 1,880 dlrs\nper tonne FOB.\n Routine sales of butter were made. March/April sold at\n4,340, 4,345 and 4,350 dlrs.\n April/May butter went at 2.27 times New York May, June/July\nat 4,400 and 4,415 dlrs, Aug/Sept at 4,351 to 4,450 dlrs and at\n2.27 and 2.28 times New York Sept and Oct/Dec at 4,480 dlrs and\n2.27 times New York Dec, Comissaria Smith said.\n Destinations were the US, Covertible currency areas,\nUruguay and open ports.\n Cake sales were registered at 785 to 995 dlrs for\nMarch/April, 785 dlrs for May, 753 dlrs for Aug and 0.39 times\nNew York Dec for Oct/Dec.\n Buyers were the US, Argentina, Uruguay and convertible\ncurrency areas.\n Liquor sales were limited with March/April selling at 2,325\nand 2,380 dlrs, June/July at 2,375 dlrs and at 1.25 times New\nYork July, Aug/Sept at 2,400 dlrs and at 1.25 times New York\nSept and Oct/Dec at 1.25 times New York Dec, Comissaria Smith\nsaid.\n Total Bahia sales are currently estimated at 6.13 mln bags\nagainst the 1986/87 crop and 1.06 mln bags against the 1987/88\ncrop.\n Final figures for the period to February 28 are expected to\nbe published by the Brazilian Cocoa Trade Commission after\ncarnival which ends midday on February 27.\n Reuter\n' >>>

I removed the following characters from the XML source to clear it ...

 (py26_default)[ mpenning@Bucksnort ~]$ python fix_xml.py foo.xml bar.xml 343 &#5; 347 &#5; 351 &#5; 359 &#22; 364 &#22; 369 &#1; 378 &#31; 444 &#2; 3393 &#3; (py26_default)[ mpenning@Bucksnort ~]$

Keep in mind that there are other ways to do this ... lxml.soupparser also cleans up bad XML. lxml.soupparser

 from lxml.html import soupparser from StringIO import StringIO try: parser = XMLParser(ns_clean=True, recover=True) tree = ET.parse(StringIO(text), parser) except UnicodeDecodeError: tree = soupparser.parse(StringIO(text))

Xml cleared

 <!DOCTYPE lewis SYSTEM "lewis.dtd"> <REUTERS TOPICS="YES" LEWISSPLIT="TRAIN" CGISPLIT="TRAINING-SET" OLDID="5544" NEWID="1"> <DATE>26-FEB-1987 15:01:01.79</DATE> <TOPICS><D>cocoa</D></TOPICS> <PLACES><D>el-salvador</D><D>usa</D><D>uruguay</D></PLACES> <PEOPLE></PEOPLE> <ORGS></ORGS> <EXCHANGES></EXCHANGES> <COMPANIES></COMPANIES> <UNKNOWN> CT f0704reute uf BC-BAHIA-COCOA-REVIEW 02-26 0105</UNKNOWN> <TEXT> <TITLE>BAHIA COCOA REVIEW</TITLE> <DATELINE> SALVADOR, Feb 26 - </DATELINE><BODY>Showers continued throughout the week in the Bahia cocoa zone, alleviating the drought since early January and improving prospects for the coming temporao, although normal humidity levels have not been restored, Comissaria Smith said in its weekly review. The dry period means the temporao will be late this year. Arrivals for the week ended February 22 were 155,221 bags of 60 kilos making a cumulative total for the season of 5.93 mln against 5.81 at the same stage last year. Again it seems that cocoa delivered earlier on consignment was included in the arrivals figures. Comissaria Smith said there is still some doubt as to how much old crop cocoa is still available as harvesting has practically come to an end. With total Bahia crop estimates around 6.4 mln bags and sales standing at almost 6.2 mln there are a few hundred thousand bags still in the hands of farmers, middlemen, exporters and processors. There are doubts as to how much of this cocoa would be fit for export as shippers are now experiencing dificulties in obtaining +Bahia superior+ certificates. In view of the lower quality over recent weeks farmers have sold a good part of their cocoa held on consignment. Comissaria Smith said spot bean prices rose to 340 to 350 cruzados per arroba of 15 kilos. Bean shippers were reluctant to offer nearby shipment and only limited sales were booked for March shipment at 1,750 to 1,780 dlrs per tonne to ports to be named. New crop sales were also light and all to open ports with June/July going at 1,850 and 1,880 dlrs and at 35 and 45 dlrs under New York july, Aug/Sept at 1,870, 1,875 and 1,880 dlrs per tonne FOB. Routine sales of butter were made. March/April sold at 4,340, 4,345 and 4,350 dlrs. April/May butter went at 2.27 times New York May, June/July at 4,400 and 4,415 dlrs, Aug/Sept at 4,351 to 4,450 dlrs and at 2.27 and 2.28 times New York Sept and Oct/Dec at 4,480 dlrs and 2.27 times New York Dec, Comissaria Smith said. Destinations were the US, Covertible currency areas, Uruguay and open ports. Cake sales were registered at 785 to 995 dlrs for March/April, 785 dlrs for May, 753 dlrs for Aug and 0.39 times New York Dec for Oct/Dec. Buyers were the US, Argentina, Uruguay and convertible currency areas. Liquor sales were limited with March/April selling at 2,325 and 2,380 dlrs, June/July at 2,375 dlrs and at 1.25 times New York July, Aug/Sept at 2,400 dlrs and at 1.25 times New York Sept and Oct/Dec at 1.25 times New York Dec, Comissaria Smith said. Total Bahia sales are currently estimated at 6.13 mln bags against the 1986/87 crop and 1.06 mln bags against the 1987/88 crop. Final figures for the period to February 28 are expected to be published by the Brazilian Cocoa Trade Commission after carnival which ends midday on February 27. Reuter </BODY></TEXT> </REUTERS>

Unknown XML Finder John Machin (fix_xml.py)

As John Machin mentions in this answer , some characters are not valid XML; he wrote this script to help find invalid XML characters.

 # coding: ascii # Find numeric character references that refer to Unicode code points # that are not valid in XML. # Get byte offsets for seeking etc in undecoded file bytestreams. # Get unicode offsets for checking against ElementTree error message, # **IF** your input file is small enough. BYTE_OFFSETS = True import sys, re, codecs fname = sys.argv[1] print fname if BYTE_OFFSETS: text = open(fname, "rb").read() else: # Assumes file is encoded in UTF-8. text = codecs.open(fname, "rb", "utf8").read() rx = re.compile("&#([0-9]+);|&#x([0-9a-fA-F]+);") endpos = len(text) pos = 0 while pos < endpos: m = rx.search(text, pos) if not m: break mstart, mend = m.span() target = m.group(1) if target: num = int(target) else: num = int(m.group(2), 16) # #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF] if not(num in (0x9, 0xA, 0xD) or 0x20 <= num <= 0xD7FF or 0xE000 <= num <= 0xFFFD or 0x10000 <= num <= 0x10FFFF): print mstart, m.group() pos = mend
+4
source

I do not know if this will help you, but I had a similar problem, and I needed to get the xml data in the element tree, and not in BeautifulSoup or lxml soupparser. I also did not want to make two passes through my xml file. So, I learned how to create my own XMLParser for ElementTree (but not cElementTree, though). Using some of Mike's code, I created an XMLParser class that can intercept character data and filter out invalid characters before passing through the parser.

Here you go:

 import xml.etree.ElementTree as ET import sys import re class MyXMLParser(ET.XMLParser): rx = re.compile("&#([0-9]+);|&#x([0-9a-fA-F]+);") def feed(self,data): m = self.rx.search(data) if m is not None: target = m.group(1) if target: num = int(target) else: num = int(m.group(2), 16) if not(num in (0x9, 0xA, 0xD) or 0x20 <= num <= 0xD7FF or 0xE000 <= num <= 0xFFFD or 0x10000 <= num <= 0x10FFFF): # is invalid xml character, cut it out of the stream print 'removing %s' % m.group() mstart, mend = m.span() mydata = data[:mstart] + data[mend:] else: mydata = data super(MyXMLParser,self).feed(mydata) parser = MyXMLParser(encoding='utf-8') xml_filename = sys.argv[1] xml_etree = ET.parse(xml_filename, parser=parser)
0
source

Source: https://habr.com/ru/post/1411421/

More articles:


All Articles