Quote:Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.
It works likes this: a robot wants to vists a Web site URL, say www.example.com/welcome.html. Before it does so, it firsts checks for www.example.com/robots.txt, and finds:
Quote:Does the site need a robots.txt file?
Quote:This gives me the IPs of bad robots I can block with my firewall or .htaccess file.