All geek questions in one place

Where can I get a list of User Agents for SEO bots?

I implement a simplified filter on how many of my unregistered site users can access. Naturally, I want to give SEO bots free board / access to most of the site.

I know this is simplistic. But it is not worth doing something more complicated. I need to compile a list of user agent names that I allow, for this I need a list of bot names starting with Googlebot (I don’t even know if this is the official spelling of Gogles web crawling bot).

Someone I need a link to a specific resource that gives the names of SEO indexing robots ?. I tried http://www.user-agents.org/ , but the granularity is not good enough - it seems to list every user agent ever created!

+4
source share
2 answers

Try this list: http://www.useragentstring.com/pages/useragentstring.php?typ=Crawler

Although the combination of Google, Yahoo, Bing, Baidu, Ask and AOL represents almost 100% of the search engine market. Therefore, I would recommend adding custom crawler agents only for your filter, and you really don't need to worry about the rest.

+8
source

You should also remember that do not view your content through the cache server, and then:

<meta name="robots" content="noarchive">

0
source

Source: https://habr.com/ru/post/1314031/

More articles:


All Articles