Geek Answers Handbook

Allow only Google CSE and disable standard Google search in ROBOTS.txt

I have a website where I use the Google Custom Search Engine. I want Google CSE to crawl my site, but I want it not to appear in the usual Google search results. I put this in a robots.txt file, hoping that google CSE bots will ignore it, since I specified the pages that I wanted Google CSE to crawl in the settings

User-agent: * Disallow: /

I think Google CSE bots should also obey robots.txt. So, is there a way to prevent my page from looking for search queries, but can I still index it for Google CSE? TIA!

+6
source share
1 answer

There is no solution to this issue that matches what you would like. I also have the same situation when I need only user search. Unfortunately, the Google crawler list does not display a specific bot for Google Custom Search. Blocking Googlebot will destroy the built-in search and custom search.

+1
source

More articles:


All Articles