Author
Message
Car Guy
Posts:422
Preferred member
Member since: 2006-11-30
:: Quote ::
Subject: Robots.txt file and site diagnostics
After not having checked my AdSense Site Diagnostics in some time, a week ago I had a look at them and noticed a list of blocked URLs, all beginning with "http:/ / webcache. googleusercontent. com/ search? q= cache:" with the "Reason Blocked" being "Robots.txt File". I then did some research and added googleusercontent. com to my robots.txt file's list of allowed spidering sites.

A few days later I checked my Site Diagnostics and saw more of the same thing, with the "Reason Blocked" still being "Robots.txt File", so I just deleted the robots.txt file. For some reason, that still hasn't stopped this from happening. Several days after deleting the robots.txt file, I'm still seeing new blocked URLs beginning with "http:/ / webcache. googleusercontent. com/ search? q= cache:" with the "Reason Blocked" being "Robots.txt File".

What should I do?
July 26, 2011 01:41AM
Ian C. Purdie
Posts:2220
Senior member
Member since: 2008-12-12
:: Quote ::
Subject: Re: Robots.txt file and site diagnostics
Quote:
What should I do?
Prolly nothing.

I did the exact same thing yesterday and the crap Google spews out is unbelievable.

As far as I can discern, Google complains about not being able to access certain directories of mine e.g. Temp, Amazon affiliate pages for books etc. and also I won't allow them to index my jpg files.

Another irksome thing is I get the message "Ads have appeared on sites not allowed". Ninety+ percent of the time they're Google's own translating sites. The latest was:

www.google.it

Yesterday in webmaster tools was a list of pages showing up 404. They're crap links to me from other sites with typos.

FWIW: According to those tools, The University of NSW has the most inward links to me with a total of 11,076
July 26, 2011 02:11AM
Car Guy
Posts:422
Preferred member
Member since: 2006-11-30
:: Quote ::
Subject: Re: Robots.txt file and site diagnostics
Thanks for the reply. Fortunately, doing nothing is something I'm pretty good at.

By the way, I forgot to mention that in my initial attempt to figure this out, I called Google's main phone number. I never did to talk to a live human, and gave up after waiting on hold for 55 minutes.
July 26, 2011 03:15AM
Harold
Posts:2494
Senior member
Member since: 2006-11-30
:: Quote ::
Subject: Re: Robots.txt file and site diagnostics
I believe doing nothing is appropriate in this case. I vaguely remember reading something about this issue, which I've also noticed in the AdSense control panel for my site. I think the block is at Google's end, not ours...
July 26, 2011 04:39AM

Sorry, you do not have permission to post/reply in this forum.