Geek Answers Handbook

Product release in errback or in case of request 302

I have a problem using scrapy

yield Request(a_url[0],
                    meta={'item': aitem}, dont_filter=True,
                    callback=self.redeem_url, errback=self.error_page)


    def redeem_url(self, response):
       item = response.request.meta['item']
       item['Click_to_Redeem_URL'] = response.url
       yield item

aitemFilled before the request a_url[0]. Sometimes I get the status 302, 404, 301 from Request, so I want, if I can’t get a 200 response from a_url[0], the request element must yield, I did not find any way, because when I had 302 scrapy retry on this request and don’t go to error_pageerrback, but if in 404 it goes to error_page, what will be the way to get the element there, because what I know in errback, I can’t get the response object, but the reject object that does not contain the element in the meta.

early

+5
source share
1 answer

You can try:

yield Request(a_url[0],
                meta={'item': aitem, 'dont_retry':1}, dont_filter=True,
                callback=self.redeem_url, 
                errback=lambda x:self.error_page(x,aitem))


def redeem_url(self, response):
   item = response.request.meta['item']
   item['Click_to_Redeem_URL'] = response.url
   yield item

dont_retry :

http://readthedocs.org/docs/scrapy/en/latest/topics/downloader-middleware.html#module-scrapy.contrib.downloadermiddleware.retry

.

+2

More articles:


All Articles