Geek Answers Handbook

How to extract a list of items using scrapely?

I use scrapely to extract data from some HTML, but I'm having difficulty retrieving a list of elements.

The scrapely github project describes just a simple example:

from scrapely import Scraper
s = Scraper()

s.train(url, data)
s.scrape(another_url)

This is good if, for example, you are trying to extract data as described:

Usage (API)

Scrapely has a powerful API, including a template format that can be edited externally, which you can use to create very capable scraper.

What follows this section is a quick example of the simplest possible use that you can run in the Python shell.

However, I am not sure how to extract the data if you find something like

Ingredientes

- 50 gr de hojas de albahaca
- 4 cucharadas (60 ml) de piñones
- 2 - 4 dientes de ajo
- 120 ml (1/2 vaso) de aceite de oliva virgen extra
- 115 gr de queso parmesano recién rallado
- 25 gr de queso pecorino recién rallado ( o queso de leche de oveja curado)

, xpath css, , .

+4
2

Scrapely . , , Python . , : (: 10 url1, : 7 url2.)

from scrapely import Scraper

s = Scraper()

url1 = 'http://www.sabormediterraneo.com/recetas/postres/leche_frita.htm'
data = {'ingreds': ['medio litro de leche',   # first and last items
  u'canela y az\xfacar para espolvorear']}
s.train(url1, data)

url2 = 'http://www.sabormediterraneo.com/recetas/cordero_horno.htm'
print s.scrape(url2)

:

[{u'ingreds': [
  u' 2 piernas o dos paletillas de cordero lechal o recental ',
  u'3 dientes de ajo',
  u'una copita de vino tinto / o / blanco',
  u'una copita de agua',
  u'media copita de aceite de oliva',
  u'or\xe9gano, perejil',
  u'sal, pimienta negra y aceite de oliva']}]

(http://www.sabormediterraneo.com/cocina/salsas6.htm) "recetas". , , . ( ).

+5

Scraply (, <ul> <ol>) - . . , HTML/, , , (<li></li>), , , , .

, , , . , case .split('\n')[3:-2] :

['- 50 gr de hojas de albahaca',
 '- 4 cucharadas (60 ml) de piñones',
 '- 2 - 4 dientes de ajo',
 '- 120 ml (1/2 vaso) de aceite de oliva virgen extra',
 '- 115 gr de queso parmesano recién rallado',
 '- 25 gr de queso pecorino recién rallado ( o queso de leche de oveja curado)']

( - ), Scraper , :

class PostprocessScraper(Scraper):

    def scrape_page_postprocess(self, page, processors=None):
        if processors == None:
            processors = {}

        result = self.scrape_page(page)
        for r in result:
            for field, items in r.items():
                if field in processors:
                    fn = processors[field]
                    r[field] = [fn(i) for i in items]

        return result

scrape_page_postprocess , . :

processors = {'ingredients': lambda s: s.split('\n')[3:-2]}
scrape_page_postprocess(page, processors)
+3

More articles:


All Articles