Geek Answers Handbook

Why are my Scrapy I / O processors not working?

I am trying to complete this tutorial .

I want my field to deschave a single line, normalized to separate spaces, and in uppercase.

dmoz_spider.py

import scrapy
from tutorial.items import DmozItem

class DmozSpider(scrapy.Spider):
    name = "dmoz"
    allowed_domains = ["dmoz.org"]
    start_urls = [
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
    ]

    def parse(self, response):
        for sel in response.xpath('//ul/li'):
            item = DmozItem()
            item['title'] = sel.xpath('a/text()').extract()
            item['link'] = sel.xpath('a/@href').extract()
            item['desc'] = sel.xpath('text()').extract()
            yield item

I tried to declare I / O processors according to http://doc.scrapy.org/en/latest/topics/loaders.html#declaring-input-and-output-processors

items.py

import scrapy
from scrapy.loader.processors import MapCompose, Join

class DmozItem(scrapy.Item):
    title = scrapy.Field()
    link = scrapy.Field()
    desc = scrapy.Field(
        input_processor=MapCompose(
            lambda x: ' '.join(x.split()),
            lambda x: x.upper()
        ),
        output_processor=Join()
    )

However, my conclusion is still obtained as follows.

{'desc': ['\r\n\t\r\n                                ',
          ' \r\n'
          '\t\t\t\r\n'
          '                                - By David Mertz; Addison Wesley. '
          'Book in progress, full text, ASCII format. Asks for feedback. '
          '[author website, Gnosis Software, Inc.]\r\n'
          '                                \r\n'
          '                                ',
          '\r\n                                '],
 'link': ['http://gnosis.cx/TPiP/'],
 'title': ['Text Processing in Python']}

What am I doing wrong?

I am using Python 3.5.1 and Scrapy 1.1.0

I post all of my code here: https://github.com/prashcr/scrapy_tutorial so you can try and modify it as you wish.

+4
1

, : Item.

, / ( ?), input_processor ItemLoader, , .

DmozItem :

from scrapy.loader import ItemLoader

class DmozSpider(scrapy.Spider):
    # ...

    def parse(self, response):
        for sel in response.xpath('//ul/li'):
            loader = ItemLoader(DmozItem(), selector=sel)
            loader.add_xpath('title', 'a/text()')
            loader.add_xpath('link', 'a/@href')
            loader.add_xpath('desc', 'text()')
            yield loader.load_item()

, input_processor output_processor Item Field .

Item:

class DmozItem(scrapy.Item):
    title = scrapy.Field()
    link = scrapy.Field()
    desc = scrapy.Field()


class MyItemLoader(ItemLoader):
    desc_in = MapCompose(
        lambda x: ' '.join(x.split()),
        lambda x: x.upper()
    )

    desc_out = Join()

:

def parse(self, response):
    for sel in response.xpath('//ul/li'):
        loader = MyItemLoader(DmozItem(), selector=sel)
        loader.add_xpath('title', 'a/text()')
        loader.add_xpath('link', 'a/@href')
        loader.add_xpath('desc', 'text()')
        yield loader.load_item()
+4

More articles:


All Articles