Thread: what the hell is Robots.txt?
View Single Post
  #7  
Old 04-11-2012, 11:52 PM
Truth Truth is offline
Banned


Join Date: Nov 2011
Location: h00t
Posts: 842
Send a message via ICQ to Truth Send a message via AIM to Truth Send a message via MSN to Truth Send a message via Yahoo to Truth
Default
robots.txt is just a courtesy for bots, but the rules defined dont have to be followed. if its on da web its on da web. but for legit bots like googlebot and archive.org, and not just spammy content scrapers, then ye u outta luck pal