Slow down web site

Question

Is there a way to make a spider slow its website promotion? Anything that can be placed in headers or robots.txt?

I thought I remembered reading something about it, but I can’t find anything now.

Answer 1

If you mean Google, you can limit the speed with which Google uses your site using your Google webmaster account (Google Webmaster Tools ).

There is also this that can be placed in a robots.txt file

User-agent: *
Crawl-delay: 10

. , robots.txt, , YMMV.

Answer 2

Google - Google (. ), Yahoo! Bing Crawl-delay robots.txt:

http://en.wikipedia.org/wiki/Robots.txt#Nonstandard_extensions

, , , (, , , , ).

Answer 3

. - Perl. , Slashdot, .

Answer 4

I don’t think robots will do anything but allow or ban. Most search engines let you customize how they index your site.

For example: Bing and Google

If you have a specific agent that is causing problems, you can either block it or see if you can configure it.