Geek Answers Handbook

How to detect scanners in Asp.net MVC

I use an action filter that checks which version of the browser is used on my site if in an older browser I place a div at the top asking for an update. I don't want web crawlers to receive a div message, so I implemented HttpBrowserCapabilitiesBase.Crawler and it looks like it works for Google, but Bing and others don't seem to register as crawlers. It’s strange for a Microsoft product not to notice Bing as a crawler!

Is there a way to add user agents to the crawler property or something like that?

Thanks!

Edited: I am using asp.net mvc 3, it looks like I need to use .Browser (?) Files. Does anyone know a comprehensive set of .Browser files for Bing and other scanners?

+4
source share
2 answers

You probably need to update the browscap.ini file since the one that comes with IIS is probably old. You can get a new one at one of the following URLs:

browscap.ini usually lives in: c: \ windows \ system32 \ inetsrv \ browscap.ini

+5
source

We do not use MVC, but do the following:

Regex.IsMatch(Request.UserAgent, @"bot|crawler|baiduspider|80legs|ia_archiver|voyager|curl|wget|yahoo! slurp|mediapartners-google", RegexOptions.IgnoreCase);

Other options in my answer here:

Finding honest web crawlers

+2
source

More articles:


All Articles