Monday, January 23, 2012

The Web Robots Pages

The Web Robots Pages: "It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:"

'via Blog this'

No comments:

Post a Comment