what the hell is Robots.txt?

[Ashimar]

I heard the Wayback Machine is good to look up old websites that arent around anymore and were archived. Well apparently you cant view some because of "Robots.txt" ...

Trying to learn about it the page just confuses me. i just want to see the goddamn webpage. Anyone know a way around it? what the hell is this shit?

the link was EQ Easter eggs over at http://wayback.archive.org/web/*/htt...asterquest.htm