How to restrict site indexing

Question

I know this question has been asked many times, but I want to be more specific.

I have a development domain and moved the site there to a subfolder. Let them talk:

http://www.example.com/ 

To:

 http://www.example.com/backup 

Therefore, I want the subfolder not to be indexed by search engines. I put robots.txt with the following content in a subfolder (can I put it in a subfolder or should it always be in the root, because I want the content in the root to be visible to search engines):

 User-agent: * Disallow: / 

Maybe I need to replace it and paste it in the root:

 User-agent: * Disallow: /backup 

Another thing, I read somewhere that some robots do not respect the robots.txt file, so just putting the .htaccess file in the / backup folder does this work?

 Order deny,allow Deny from all 

Any ideas?

Answer 1

This will prevent indexing of this directory:

 User-agent: * Disallow: /backup/ 

In addition, the robots.txt file must be placed in the root of your domain, so in this case the file will be placed where you can access it in your browser by going to http://example.com/robots.txt

As an aside, you might consider creating a subdomain for your development site, something like http://dev.example.com . This will allow you to completely separate the dev files from your production environment, as well as provide a more accurate match for your environment.

For example, any absolute paths to JavaScript files, CSS, images, or other resources may not work the same way with dev for production, and this may lead to some problems in the future.

For more information on configuring this file, see robotstxt.org . Good luck

As a last and last note, Google Webmaster Tools contains a section where you can see what is blocked by the robots.txt file :

To find out which Google URLs were blocked from crawling, go to the Blocked URLs page of the Webmaster Tools Health section.

I highly recommend that you use this tool, since a misconfigured robots.txt file can significantly affect the performance of your site.