I have long been thinking about banning every crawler except Ask, Google, Microsoft and Yahoo! from my site.
The rationale for this is that I have never seen any traffic generated by any of the other web crawlers.
My questions:
- Is there any reason not to?
- Has anyone done this?
- Have you noticed any negative consequences?
Update:
So far I have used the blacklist: if I don't like the finder, I add them to the ban list.
However, I am not a fan of blacklists, because this is an endless story: there are always more scanners there.
I don’t really care about real ugly distorting searchers, they are detected and blocked automatically. (and they usually do not request a robots.txt file :)
However, many seekers are by no means mistaken; they simply do not create any value for me / my clients.
There are, for example, several crawlers that use a website that claims to be Google's next; Only better. I have never seen the traffic coming from them, and I am skeptical of them, becoming better than any of the four search engines mentioned above.
2:
, , 100 - (= , ). 52% - .
60% robots.txt, 40% (21% )
robots.txt. ( Ask, Google, Microsoft Yahoo!)
, , , , , , 12% - 17%.