How to send message data to start_urls scrapy spider

Question

I want to crawl a website that only supports sending data. I want to send request parameters in the mail data in all requests. How to achieve this?

Answer 1

POST requests can be made using scrapy Request or FormRequest classes.

Also consider using start_requests() instead of the start_urls property.

Example:

 from scrapy.http import FormRequest class myspiderSpider(Spider): name = "myspider" allowed_domains = ["www.example.com"] def start_requests(self): return [ FormRequest("http://www.example.com/login", formdata={'someparam': 'foo', 'otherparam': 'bar'}, callback=self.parse) ] 

Hope this helps.