Geek Answers Handbook

Should I get rid of bots visiting my site?

I noticed on my trackers that bots are visiting my ALOT site. Should I change or modify the robots.txt file or change something? Not sure if this is good because they index or what?

+5
source share
5 answers

Should I change or modify the robots.txt file or change something?

Depends on the bot. Some bots dutifully ignore robots.txt. We had a similar problem 18 months ago with the Google AD bot because our client bought Soooo a lot of ads. Google AD bots (as documented) ignore wildcard (*) exceptions (*), but listen for explicit ignorements.

, , robots.txt . , , .

.

, , ?

//. . , HTTP- UserAgents. , - .

, , UserAgent, 403 . , , :

  • White-List UserAgents - . , . .
  • IP - http IP- . , DOS'd ( ), .
+4

, robots.txt , . . http://www.codeplex.com/urlrewriter, , , .

+4

- robots.txt. - mod_security ( Apache). .

+3

, .htaccess, . . : http://spamhuntress.com/2006/02/13/another-hungry-java-bot/

Java, ,

- SetEnvIfNoCase ^ Java/1. javabot =
User-Agent SetEnvIfNoCase ^ Java1. javabot =
env = javabot

. 403 :)

+2

- , , . , - .

After trying to recapture some of them for some time, but the bots simply continued to change their recognizable characteristics. We are done with the following strategy:

For each session on the server, we determined whether the user was pressing too fast at any time. After a given number of retries, we set the isRobot flag to true and simply reduce the response speed in this session by adding more beds. We did not tell the user in any way, since he was just starting a new session in this case.

+2
source

More articles:


All Articles