How to retry for 404 link not found in scrapy?

Question

Some sites give a 404 error temporarily. But I insert it into the browser, it works. How to say that scrapy repeats the links of the 404 status code 5 times.

Answer 1

There are two Scrapy options you need:

• RETRY_HTTP_CODES : you must override the default value in your project to enable 404
• RETRY_TIMES : just set it to 5