Geek Answers Handbook

Start screening for Flask

I want to create a crawler that displays the URL of a webpage and returns the result to the webpage. Right now I'm starting to tear myself away from the terminal and save the answer in a file. How to start a crawler when any input is sent to Flask, process and return a response?

+5
source share
1 answer

You need to create a CrawlerProcess inside your Flask application and start the crawl programmatically. See docs .

import scrapy from scrapy.crawler import CrawlerProcess class MySpider(scrapy.Spider): # Your spider definition ... process = CrawlerProcess({ 'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)' }) process.crawl(MySpider) process.start() # The script will block here until the crawl is finished

Before moving on to the project, I advise you to look into the Python task queue (for example, rq ). This will allow you to run Scrapy crawls in the background and the Flask application will not hang while scrapes is running.

+4
source

More articles:


All Articles