How can I fix the problem "Googlebot cannot access your site"?

Question

I just get a message about

"Over the past 24 hours, Googlebot has detected 1 error while trying to access your robots.txt. To ensure that we did not crawl all the pages listed in this file, we delayed our crawl. The total error of your robots.txt site is 100, 0%. For more information about these errors, see Webmaster Tools. "

I searched it and told me to add a robots.txt file on my website

And when I test the robots.txt file in Google’s webmaster tools, the robots.txt file simply cannot be extracted.

I thought robots.txt was blocked by my site, but when I check it, it is allowed by GWT.

' http://momentcamofficial.com/robots.txt ' And here is the contents of robots.txt: User agent: * Disallow:

So why robots.txt cannot be delivered by Google? What I missed ... Can someone help me?

Answer 1

Before Googlebot crawls your site, it consults your robots.txt file to determine whether your Google site blocks any pages or URLs. If the robots.txt file exists but is not available (in other words, if it does not return an HTTP status code of 200 or 404), it’s good to postpone our crawl, and not risk URLs that you do not need to crawl. When this happens, Googlebot will return to your site and crawl it as soon as we can successfully access your robots.txt file.

As you know, robots.txt is optional, so you do not need to do it, just make sure that your host has sent only 200 or 404 http status.

Answer 2

I had a situation where Google Bot did not retrieve data, but I could see a valid robots.txt file in my browser.

The problem turned out to be that I redirected the entire site (including robots.txt) to https, but Google did not seem to like it. Therefore, I excluded the robots.txt file from the redirect.

RewriteEngine On RewriteCond %{HTTPS} off RewriteCond %{REQUEST_FILENAME} !robots\.txt RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L] 

Additional Blog Information

Answer 3

You have the wrong content in the robots.txt file, change it to:

 User-agent: * Allow: / 

And make sure everyone has read permissions for the file.