Why your site NEEDS a robots.txt file

GoogleGroup’s JohnMu recommends in this thread “Site Disappeared from Google & Reconsideration.Request – Crawling, indexing, and ranking | Google Groups” that “When we can’t access the robots.txt properly (when it times out like this or is generally unreachable), we tend not to crawl the site at all just to be safe.”

So make sure that a) you have a robots.txt file and b) it’s accessible, otherwise your site may not be crawled by Googlebot at all.

Leave a Reply