Geek Answers Handbook

Optimize RSS parsing in App Engine to prevent high CPU warnings

I pull some RSS feeds into the data store in App Engine to serve the iPhone application. I use cron to schedule RSS updates every x minutes. Each task processes only one RSS feed (which contains 15-20 elements). I often get warnings about high CPU usage in the App Engine toolbar, so I'm looking for ways to optimize my code.

I am currently using minidom (since it is already present in App Engine), but I suspect it is not very effective!

Here is the code:

 dom = minidom.parseString(urlfetch.fetch(url).content)
    if dom:
        items = []
        for node in dom.getElementsByTagName('item'):
            item = RssItem(
                key_name = self.getText(node.getElementsByTagName('guid')[0].childNodes),
                title = self.getText(node.getElementsByTagName('title')[0].childNodes),
                description = self.getText(node.getElementsByTagName('description')[0].childNodes),
                modified = datetime.now(),
                link = self.getText(node.getElementsByTagName('link')[0].childNodes),
                categories = [self.getText(category.childNodes) for category in node.getElementsByTagName('category')]
            );
            items.append(item);
        db.put(items);

def getText(self, nodelist):
    rc = ''
    for node in nodelist:
        if node.nodeType == node.TEXT_NODE:
            rc = rc + node.data
    return rc

Not much happens, but scripts often take 2-6 seconds in processor time, which seems too redundant to cyclize 20ish elements and read several attributes.

, ? - , ? - ( App Engine), , RSS ?

+5
5

, , superfeedr

superfeedr.com. / . ( 15 ) / .. pubsubhubbub, ! , pubsubhubbub, .

,

video , pubsubhubbub. , - , Universal Feedparser, . SAX (14:10 , ), , . , pubsubhubbub code, , .

+4

ElementTree Universal Feed Parser , . ElementTree stdlib Python 2.5, App Engine.

+1

, , . , . , , , . appengine, , .

superfeedr www.newsfacet.com, , superfeedr , rss . , 10 11 , .

+1

PubSubHubbub , - , , hubbub App Engine .

+1

, , , . , RSS- REAL .

, RDF/RSS/ATOM , cr * p . , .

The Parser universal feed is fully functional, at least from what I saw while viewing documents. I did not use it because I wrote my aggregators in Ruby and had different needs, but I knew about it and was considering it for a Python based solution.

0
source

More articles:


All Articles