Author
Message
GegaBit
Posts:3311
Senior member
Member since: 2006-11-30
:: Quote ::
Subject: The day Google Bot Stopped Crawling
My site gets over 10,000 daily Google Bot visits including both Googlebot and AdSense Media Partners bot, I monitor the daily crawl rate and consider it a sign of health.

So when for a full 24 hours my site received zero Google bot visits I became very concerned, I scrambled to check that:

- Traffic was normal
- Robots.txt and .htaccess files were unmodified
- Google Webmaster Tools reported no crawl or any other error
- AdSense ads were being served and clicks earnings were being counted.
- Ranking for important terms was unchanged
- Error logs did not show the Google bot getting denied
- Reconfirmed that I really got no visits from the Google bot but was being crawled fine by all other bots.
- Even checked my email spam inbox for a message from Google.

4 am I figured out and fixed the issue, given all the above, let's see if you can figure out what the problem was.
September 15, 2009 05:26PM
Joshua
Posts:2831
Administrator
Member since: 2007-03-16
:: Quote ::
Subject: Re: The day Google Bot Stopped Crawling
Does that mean I'll have to stay up until 4AM?

Did you create the problem yourself in the first place?
September 15, 2009 05:56PM
James
Posts:1757
Moderator
Member since: 2006-11-29
:: Quote ::
Subject: Re: The day Google Bot Stopped Crawling
Sorry GB - I give up!!
September 15, 2009 06:19PM
GegaBit
Posts:3311
Senior member
Member since: 2006-11-30
:: Quote ::
Subject: Re: The day Google Bot Stopped Crawling
Yes Jo it was something I did.
Come on James, try harder.
September 15, 2009 06:30PM
Joshua
Posts:2831
Administrator
Member since: 2007-03-16
:: Quote ::
Subject: Re: The day Google Bot Stopped Crawling
I don't know if this is supposed to be a YES/NO quiz. But did you do 'that something' just before the bots stopped visiting?
September 15, 2009 06:47PM
GegaBit
Posts:3311
Senior member
Member since: 2006-11-30
:: Quote ::
Subject: Re: The day Google Bot Stopped Crawling
Yes, I caused the problem a day earlier.
September 15, 2009 07:09PM
Joshua
Posts:2831
Administrator
Member since: 2007-03-16
:: Quote ::
Subject: Re: The day Google Bot Stopped Crawling
We'll I can't imagine Google reacting on on-page factores within one day (like noindex, nofollow etc.) to slow down crawling, also I don't see any option in webmastertools to tell Google to stop crawling.

I'm inclined to exclude on-page factors and Google property tools.

Is it a server side created problem?
September 15, 2009 07:29PM
GegaBit
Posts:3311
Senior member
Member since: 2006-11-30
:: Quote ::
Subject: Re: The day Google Bot Stopped Crawling
yes
September 15, 2009 08:06PM
Joshua
Posts:2831
Administrator
Member since: 2007-03-16
:: Quote ::
Subject: Re: The day Google Bot Stopped Crawling
hosts or localhost file(s)?
September 15, 2009 08:29PM
Pengi
Posts:3345
Senior member
Member since: 2006-12-17
:: Quote ::
Subject: Re: The day Google Bot Stopped Crawling
Did you tell the bots not to visit you somehow?
September 15, 2009 08:36PM
Joshua
Posts:2831
Administrator
Member since: 2007-03-16
:: Quote ::
Subject: Re: The day Google Bot Stopped Crawling
Quote:
Did you tell the bots not to visit you somehow?

From my and GB's previous posts you can conclude he didn't. Either they are blocked before entering his logs, or redirected (and maybe showing up in another log) or send into a black-hole.

I'd admit guessing from this point is difficult because I can't see any of the files concerned nor do I know anything about his server configuration.

So still my question:
hosts or localhost file(s)?
September 15, 2009 09:14PM
GegaBit
Posts:3311
Senior member
Member since: 2006-11-30
:: Quote ::
Subject: Re: The day Google Bot Stopped Crawling
Jo is close

A day earlier I wanted to firewall block all the IPs of Spry hosting company from accessing my site among other IP ranges I was planning to block..

I blocked 66.249.16.0/11 instead of 66.249.16.0/19

/11 means 32 class B
/19 means 32 class C

So instead of only blocking 8,000 I blocked 2 Million IPs including Google's IP Range on the firewall which prevented any access denied errors being logged on the server.

Took me over 12 hours of thinking to figure out what happened and thought it might amuse you.
September 15, 2009 09:16PM
Joshua
Posts:2831
Administrator
Member since: 2007-03-16
:: Quote ::
Subject: Re: The day Google Bot Stopped Crawling
Believe it or not. But my next question would've been firewall blocking.

If that would not have been the case then any httpd config file changes (if you're running apache).

(Did I win the contest.)
September 15, 2009 09:27PM
GegaBit
Posts:3311
Senior member
Member since: 2006-11-30
:: Quote ::
Subject: Re: The day Google Bot Stopped Crawling
Yeah you did Jo

My line of thinking was in that order:

1- It was a Google ban, checked forums, previous days logs referrers, Google webmaster tools, traffic, keywords ranking

2- I was hacked, took me to checking robots and htaccess

3- Something was wrong at my hosting network

4- Something was wrong with my server config

So what does a full time webmaster do when Google drops them and there is no way to get back, go looking for a job?
September 15, 2009 09:40PM
Joshua
Posts:2831
Administrator
Member since: 2007-03-16
:: Quote ::
Subject: Re: The day Google Bot Stopped Crawling
Quote:
So what does a full time webmaster do when Google drops them and there is no way to get back, go looking for a job?

Fortunately I'm not dropped by Google. Unfortunately Adsense is not really picking up yet. So, yes, looking for a job besides my company is a real option for me. Still I hope/think/know there are a lot of companies that really messed up with their IT during the recession so that there is a lot of work to do when some moneys come available again. Just knowing when that time arrives would be invaluable.

At least I'm entering my high traffic season, so hoping for a bit better time.

Personally I'm a technology/strategy guy, so I'm not bound to the internet and should be able to find a job when necessary (have problems spelling that every time winking smiley).



Edited 1 time(s). Last edit at 09/15/2009 10:33PM by Joshua.
September 15, 2009 10:19PM

Sorry, you do not have permission to post/reply in this forum.