Debugging The Network Unreachable / Robots. txt Unreachable Error

On 01 August, 2007, I noticed a significant drop in traffic to many of my sites, including my main website. Now, most webmasters will be familiar with the occasional blip when things go wrong, so you learn to sit tight and monitor the situation. Sometimes, these things can last anywhere from a day to 2 weeks, and the last thing you DON’T want to do is react in panic and make changes that aggravate the situation. However, after 3 weeks and no sign of recovery, I started to worry. I decided to do some checking on my own. Along the way, I learnt many lessons.

The symptoms

* Google Sitemaps reports “Errors” that alternated between :

• Network unreachable: Network unreachable
  We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit.
• Network unreachable: robots. txt unreachable
  We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit.

* Checking the Diagnostics tab, I started to see many pages listed as unreachable.

* Analysis of my cached in Diagnostics revealed Google’s last cached date for my robots. txt file was was back in July 13, 2007. This would suggest that Googlebot had problems accessing my robots. txt file (and my site) after that.

* A look into my server logs showed Googlebot’s last single visit for the day was on August 10. That seemed strange when in the past, Googlebot would visit me at least once every hour on average.

* The graph of Google’s crawl rate showed Google stopped crawling my site in late July, and the second graph showed erratic download times co-inciding with Googlebot’s absence.

I believe that as a result of Googlebot being unable to access my pages, my site dropped out of the Search Engine Ranked Pages (serps), so traffic that used to come from being ranked on page one for popular terms stopped. This caused an unnerving drop in revenue.

Checking Google
As usual, my first stop when these sort of things happen is the online forums. I needed to check if this was a widespread issue that webmasters were facing. I did find a number of threads on the same problem, but not enough to verify that this was a serious bug on Google’s part. However, the small number of forum threads about a particular problem doesn’t necessarily mean that it isn’t a Google bug. After all, Google IS a machine, made up of hundreds of thousands of servers and many, many algorithms. And if your site is not a heavy-weight traffic generator, it’s unlikely that a Google bug affecting your site is going to be top-priority for the guys at Mountain View.

Checking Myself and My Sites
Once I had more or less determined that this wasn’t a widespread Google bug, I needed to make sure Googlebot didn’t stop visiting because of my own doing.

* Did Googlebot stop visiting due to a penalty?
The only thing I could think of that would possibly cause a penalty by Google was reciprocal links pages. I had stopped reciprocal linking for a long time now BUT maintained those pages out of courtesy to the sites that still kept their links to me. However, recent updates in Google’s Webmaster Guidelines suggests that reciprocal linking will hurt your site :

I wasn’t sure if my reciprocal links pages were part of the problem, but in the interest of self-preservation, I decided to remove all reciprocal link pages from all my sites.

* Was my robots. txt file okay?
First, the Google Webmaster Tools indicated to me that my robots. txt file was okay although its cached version was about 30 days old. The long lapse between the current date and the last cached date of the robots. txt file SHOULD have indicated to me that Googlebot had problems accessing the file. Robots. txt validators that I used below showed there was nothing wrong with my file :

http://www.invision-graphics.com/robotstxt_validator.html

http://validator.czweb.org/robots-txt.php

Conclusion : My robots. txt file seemed okay.

* Was my sitemap file okay?
Google’s sitemaps can seem like an unruly monster to the uninitiated. Many webmasters have reported that the Google sitemap reports have been known to be buggy at times and report errors when there were none. In any case, I used the following tools to check the validity of my sitemaps and its schemas :

http://www.validome.org/google/

http://www.smart-it-consulting.com/internet/google/submit-validate-sitemap/

http://www.xml-sitemaps.com/validate-xml-sitemap.html

https://www.google.com/webmasters/tools/docs/en/protocol.html

http://www.sitemaps.org/protocol.php

Conclusion : My sitemap.xml file seemed okay and passed validation by the tools above.

* Was my .htaccess file okay?
My investigations revealed that incorrect coding in the .htaccess file could cause unnecessary looping (meaning I could have been sending Googlebot in circles and it threw up a red flag), so this had to be checked out. There were no known validators that I could use to verify that my .htaccess file was okay so I took my problem to the forums :

• Webmaster World Forum
  I posted numerous questions on the Google Search Board and Apache Server Board.
• Google Webmaster Help
  Here, I posted my issue on the Crawling, indexing and ranking board

Since I had no experience in .htaccess coding and debugging, I had to rely on the more experienced webmasters who contribute to the forum listed above. Thankfully, all who looked at the content of my .htaccess file cleared it, saying there wasn’t any problem with my .htaccess file.

Conclusion : My .htaccess file didn’t seem to contain coding errors

* Were my pages blocking Googlebot?
It was unlikely that my pages were blocking Googlebot since I did not make major changes and Googlebot was visiting them up till the last visit on July 13 1007. However, I needed to make sure I did not inadvertently place any tag like the following in my pages :

• <META NAME=”ROBOTS” CONTENT=”NOINDEX, NOFOLLOW”>
• <META NAME=”GOOGLEBOT” CONTENT=”NOINDEX, NOFOLLOW”>

Update : Another way you may be blocking Googlebot is if you load your pages with scripts that leave Googlebot “stranded”.

Conclusion : All my pages DID NOT contain any tags that would have blocked Googlebot

Checking My Web Host Provider
The nature of the error in Google Sitemaps strongly suggested that Google was having problems accessing my site where other bots and search engines didn’t. It was a really perplexing situation. My web host initially replied that they had not blocked Googlebot, so it was left to me to find possible loopholes. I turned again to the boards. The difficult part initially was trying to find out the right question to ask. So I started by describing my problem. Then one by one, contributors made suggestions and I followed through on every one of those suggestions.

A contributor commented that using her header check tool, she found all my URLs were returning a 403 (forbidden) status. I have not placed a live link to her tool because she has since taken it offline for upgrading and maintenance. In any case, to verify the results her tool was giving me, I used another header check tool. Indeed using this tool, most of my URLs returned a “Operation Timed Out” error.

I then wondered if my site was the only one that was experiencing 403 Forbidden and time out error. I copied the URLs of all the clients listed on their “prominent clients” page and checked them with the 2 tools. Indeed, I found many of those sites returned the same 403 - Forbidden status on the first tool and timed out on the second tool. I reported this finding to my web host.

The first sign that I was probably on to something was when there was a reply from them stating they needed time to “conduct tests”. I signed up with a trial account with WatchMouse.com, a website monitoring service to see if any more timeout errors were popping up and indeed they were.

Again I reported this to my web host. Their reply stated that timeouts could be caused by many other reasons other than the servers, which I accepted. However, I noted that they had requested their Security Team to investigate the matter, which was another indication that we were on the right track.

Hurray!
Checking my stats on August 30th, I was surprised to see that the server had registered over a hundred visits from Googlebot. I immediately contacted my web host and asked if they had made any changes and they confirmed that they did. So here’s what caused the inadvertent blocking of Googlebot according to them :

Lessons Learned
In hindsight, it occured to me that modification to my .htaccess file could have caused an increase in the concurrent connections to the server. I had modified my .htaccess file to solve canonicalization problems by redirecting :

• redundant URLs to new URLs
• non-www URLs to www URLs
• index.php to root

I theorize that since these redirections involved hundreds of URLs it’s possible that when I deployed the changes in my .htaccess file in mid July, it triggered the “increase” in concurrent connections as the bots were redirected to the correct pages. In other words Googlebot attempted to make 2 connections for every page - once to the old/non-www URL and then to the new/www URL. As the concurrent connections increased, it triggered the automated mechanism that blocked Googlebot’s IP address. This in turn caused more time-out errors. The spikes in the Googlebot Download Time chart (above) indicates long download times which eventually ended in timeouts. Unfortunately, this affected one of the most important files - the robots. txt file - which every bot needs to before it accesses a site’s pages. These time-outs also made my sitemap inaccessible, so since Googlebot could not access these 2 important pages, it could not confirm my site still existed!

However, my web host’s analysis of the situation brought them to the conclusion that more of their clients (like myself) had begun to use Google Sitemaps and this is the reason for the increase in concurrent connections. Whatever the reason, at last check, Googlebot has resumed its crawl of my site and I must say it is a welcome sight.

I’ve learnt a lot as I struggled with this problem and I hope this post helps some of you who may be wondering why Googlebot has stopped visiting your site or you are experiencing the “Network unreachable: Robots. txt unreachable” error.

Popularity: 89% [?]