How can i install this? I wrote handle_httpstatus_list = [301, 302, 303], but scrapy does not follow new links, it just started saving empty page files.
PS English is not my native language. Avoid me for it.
You should also specify dont_redirect = true in your request as:
dont_redirect = true
meta = {'dont_redirect': True, "handle_httpstatus_list" : [301, 302, 303]}
Hi
Scrapy handles redirects by default, which means that RedirectMiddleware is activated in the DOWNLOADER_MIDDLEWARES setting, and REDIRECT_ENABLED set to True .
DOWNLOADER_MIDDLEWARES
REDIRECT_ENABLED
True