Welcome Home Googlebot!

After solving the Network Unreachable : Robots. txt Unreachable error, Googlebot has been back crawling my main site steadily for the past ten days. Rankings have returned to normal and more importantly, so have earnings.

The yellow colored blocks indicate when Googlebot stopped visiting. Most important are the first and third graphs. The first graph indicates that the number of pages Googlebot has resumed crawling per day has gone back to normal.

More importantly, the third graph show that Googlebot is not experiencing major fluctuations in the time taken to access my pages. If you look at the yellow section in the third graph, you can clearly see that Googlebot had serious problems trying to access my pages from the end of July and throughout August. This graph should have been the first indicator to me that something was wrong. I know now that the spikes in the third graph were time-outs that Googlebot encountered. When trying to find the source of the problem, I used the following tools to check the header status of my site :

http://www.asymptoticdesign.com/aux/header_etal.cgi
http://www.seoconsultants.com/tools/headers/

The first tool returned many 403 Forbidden errors for my pages and the second tool returned time-out errors. I did not see any 403 errors in my server logs because when a bot gets flagged with a 403 Forbidden status, it assumes that they DO NOT have clearance to crawl the pages and they just leave. This made it all the more difficult to find the source of the problem.

Knowing what I know now, Google Webmaster Tools have become my best friend.

One other good thing seems to have resulted from this problem. With Googlebot returning, it seems to be doing a fresh indexing of my site and the links being reported in my sitemap reports are up to date. I believe modifications to my .htaccess file to solve canonical issues also helped. I expect that in future, if Google decides to maintain their PR ranking system, having canonicalized my URLs will stop PR from being split between my root and index homepage and also internal pages.

Popularity: 7% [?]