My site gets over 10,000 daily Google Bot visits including both Googlebot and AdSense Media Partners bot, I monitor the daily crawl rate and consider it a sign of health.
So when for a full 24 hours my site received zero Google bot visits I became very concerned, I scrambled to check that:
- Traffic was normal
- Robots.txt and .htaccess files were unmodified
- Google Webmaster Tools reported no crawl or any other error
- AdSense ads were being served and clicks earnings were being counted.
- Ranking for important terms was unchanged
- Error logs did not show the Google bot getting denied
- Reconfirmed that I really got no visits from the Google bot but was being crawled fine by all other bots.
- Even checked my email spam inbox for a message from Google.
4 am I figured out and fixed the issue, given all the above, let's see if you can figure out what the problem was.
We'll I can't imagine Google reacting on on-page factores within one day (like noindex, nofollow etc.) to slow down crawling, also I don't see any option in webmastertools to tell Google to stop crawling.
I'm inclined to exclude on-page factors and Google property tools.
So what does a full time webmaster do when Google drops them and there is no way to get back, go looking for a job?
Fortunately I'm not dropped by Google. Unfortunately Adsense is not really picking up yet. So, yes, looking for a job besides my company is a real option for me. Still I hope/think/know there are a lot of companies that really messed up with their IT during the recession so that there is a lot of work to do when some moneys come available again. Just knowing when that time arrives would be invaluable.
At least I'm entering my high traffic season, so hoping for a bit better time.
Personally I'm a technology/strategy guy, so I'm not bound to the internet and should be able to find a job when necessary (have problems spelling that every time ).
Edited 1 time(s). Last edit at 09/15/2009 10:33PM by Joshua.