Project 1999 - what the hell is Robots.txt?

Project 1999 (/forums/index.php)

- Rants and Flames (/forums/forumdisplay.php?f=30)

- - what the hell is Robots.txt? (/forums/showthread.php?t=71609)

what the hell is Robots.txt?

I heard the Wayback Machine is good to look up old websites that arent around anymore and were archived. Well apparently you cant view some because of "Robots.txt" ...

Trying to learn about it the page just confuses me. i just want to see the goddamn webpage. Anyone know a way around it? what the hell is this shit?

the link was EQ Easter eggs over at http://wayback.archive.org/web/*/htt...asterquest.htm

do you even read the text under the error, you gigantic fucktard?

http://en.wikipedia.org/wiki/Robots_exclusion_standard

#
# robots.txt for http://www.wikipedia.org/ and friends
#
# Please note: There are a lot of pages on this site, and there are
# some misbehaved spiders out there that go _way_ too fast. If you're
# irresponsible, your access to the site may be blocked.
#

# advertising-related bots:
User-agent: Mediapartners-Google*
Disallow: /

# Wikipedia work bots:
User-agent: IsraBot
Disallow:

User-agent: Orthogaffe
Disallow:

# Crawlers that are kind enough to obey, but which we'd rather not have
# unless they're feeding search engines.
User-agent: UbiCrawler
Disallow: /

User-agent: DOC
Disallow: /

User-agent: Zao
Disallow: /

# Some bots are known to be trouble, particularly those designed to copy
# entire sites. Please obey robots.txt.
User-agent: sitecheck.internetseer.com
Disallow: /

User-agent: Zealbot
Disallow: /

User-agent: MSIECrawler
Disallow: /

User-agent: SiteSnagger
Disallow: /

User-agent: WebStripper
Disallow: /

User-agent: WebCopier
Disallow: /

User-agent: Fetch
Disallow: /

User-agent: Offline Explorer
Disallow: /

User-agent: Teleport
Disallow: /

User-agent: TeleportPro
Disallow: /

User-agent: WebZIP
Disallow: /

User-agent: linko
Disallow: /

User-agent: HTTrack
Disallow: /

User-agent: Microsoft.URL.Control
Disallow: /

User-agent: Xenu
Disallow: /

User-agent: larbin
Disallow: /

User-agent: libwww
Disallow: /

User-agent: ZyBORG
Disallow: /

User-agent: Download Ninja
Disallow: /

#
# Sorry, wget in its recursive mode is a frequent problem.
# Please read the man page and use it properly; there is a
# --wait option you can use to set the delay between hits,
# for instance.
#
User-agent: wget
Disallow: /

#
# The 'grub' distributed client has been *very* poorly behaved.
#
User-agent: grub-client
Disallow: /

#
# Doesn't follow robots.txt anyway, but...
#
User-agent: k2spider
Disallow: /

#
# Hits many times per second, not acceptable
# http://www.nameprotect.com/botinfo.html
User-agent: NPBot
Disallow: /

# A capture bot, downloads gazillions of pages with no public benefit
# http://www.webreaper.net/
User-agent: WebReaper
Disallow: /

# Don't allow the wayback-maschine to index user-pages
#User-agent: ia_archiver
#Disallow: /wiki/User
#Disallow: /wiki/Benutzer

#
# Friendly, low-speed bots are welcome viewing article pages, but not
# dynamically-generated pages please.
#
# Inktomi's "Slurp" can read a minimum delay between hits; if your
# bot supports such a thing using the 'Crawl-delay' or another
# instruction, please let us know.
#
User-agent: *
Disallow: /w/
Disallow: /trap/
Disallow: /wiki/Especial:Search
Disallow: /wiki/Especial%3ASearch
Disallow: /wiki/Special:Collection
Disallow: /wiki/Spezial:Sammlung
Disallow: /wiki/Special:Random
Disallow: /wiki/Special%3ARandom
Disallow: /wiki/Special:Search
Disallow: /wiki/Special%3ASearch
Disallow: /wiki/Spesial:Search
Disallow: /wiki/Spesial%3ASearch
Disallow: /wiki/Spezial:Search
Disallow: /wiki/Spezial%3ASearch
Disallow: /wiki/Specjalna:Search
Disallow: /wiki/Specjalna%3ASearch
Disallow: /wiki/Speciaal:Search
Disallow: /wiki/Speciaal%3ASearch
Disallow: /wiki/Speciaal:Random
Disallow: /wiki/Speciaal%3ARandom
Disallow: /wiki/Speciel:Search
Disallow: /wiki/Speciel%3ASearch
Disallow: /wiki/Speciale:Search
Disallow: /wiki/Speciale%3ASearch
Disallow: /wiki/Istimewa:Search
Disallow: /wiki/Istimewa%3ASearch
Disallow: /wiki/Toiminnot:Search
Disallow: /wiki/Toiminnot%3ASearch
#
# ar:
Disallow: /wiki/%D8%AE%D8%A7%D8%B5:Search
Disallow: /wiki/%D8%AE%D8%A7%D8%B5%3ASearch
#
# de:
# http://bugzilla.wikimedia.org/show_bug.cgi?id=4937
# sensible deletion and meta user discussion pages:
Disallow: /wiki/Wikipedia:L%C3%B6schkandidaten/
Disallow: /wiki/Wikipedia:Löschkandidaten/
Disallow: /wiki/Wikipedia:Vandalensperrung/
Disallow: /wiki/Wikipedia:Benutzersperrung/
Disallow: /wiki/Wikipedia:Vermittlungsausschuss/
Disallow: /wiki/Wikipedia:Administratoren/Probleme/
Disallow: /wiki/Wikipedia:Adminkandidaturen/
Disallow: /wiki/Wikipedia:Qualitätssicherung/
Disallow: /wiki/Wikipedia:Qualit%C3%A4tssicherung/
# Search- and random-page
Disallow: /wiki/Spezial:Suche
Disallow: /wiki/Special:Suche
Disallow: /wiki/Spezial:Zufällige_Seite
Disallow: /wiki/Spezial:Zuf%C3%A4llige_Seite
Disallow: /wiki/Special:Zufällige_Seite
Disallow: /wiki/Special:Zuf%C3%A4llige_Seite
# 4937#5
Disallow: /wiki/Wikipedia:Vandalismusmeldung/
Disallow: /wiki/Wikipedia:Gesperrte_Lemmata/
Disallow: /wiki/Wikipedia:Löschprüfung/
Disallow: /wiki/Wikipedia:L%C3%B6schprüfung/
Disallow: /wiki/Wikipedia:Administratoren/Notizen/
Disallow: /wiki/Wikipedia:Schiedsgericht/Anfragen/
Disallow: /wiki/Wikipedia:L%C3%B6schpr%C3%BCfung/
# http://bugzilla.wikimedia.org/show_bug.cgi?id=12111
Disallow: /wiki/Wikipedia:Checkuser/
Disallow: /wiki/Wikipedia_Diskussion:Checkuser/
Disallow: /wiki/Wikipedia_Diskussion:Adminkandidaturen/
# https://bugzilla.wikimedia.org/show_bug.cgi?id=13961
Disallow: /wiki/Wikipedia:Spam-Blacklist-Log
Disallow: /wiki/Wikipedia%3ASpam-Blacklist-Log
Disallow: /wiki/Wikipedia_Diskussion:Spam-Blacklist-Log
Disallow: /wiki/Wikipedia_Diskussion%3ASpam-Blacklist-Log
#
# Folks get annoyed when VfD discussions end up the number 1 google hit for
# their name. See bugzilla bug #4776
# en:
Disallow: /wiki/Wikipedia:Articles_for_deletion/
Disallow: /wiki/Wikipedia%3AArticles_for_deletion/
Disallow: /wiki/Wikipedia:Votes_for_deletion/
Disallow: /wiki/Wikipedia%3AVotes_for_deletion/
Disallow: /wiki/Wikipedia:Pages_for_deletion/
Disallow: /wiki/Wikipedia%3APages_for_deletion/
Disallow: /wiki/Wikipedia:Miscellany_for_deletion/
Disallow: /wiki/Wikipedia%3AMiscellany_for_deletion/
Disallow: /wiki/Wikipedia:Miscellaneous_deletion/
Disallow: /wiki/Wikipedia%3AMiscellaneous_deletion/
Disallow: /wiki/Wikipedia:Copyright_problems
Disallow: /wiki/Wikipedia%3ACopyright_problems
Disallow: /wiki/Wikipedia:Protected_titles/
Disallow: /wiki/Wikipedia%3AProtected_titles/
# https://bugzilla.wikimedia.org/show_bug.cgi?id=13398
Disallow: /wiki/Wikipedia:WikiProject_Spam/
Disallow: /wiki/Wikipedia%3AWikiProject_Spam/
# https://bugzilla.wikimedia.org/show_bug.cgi?id=14075
Disallow: /wiki/MediaWiki:Spam-blacklist
Disallow: /wiki/MediaWiki%3ASpam-blacklist
Disallow: /wiki/MediaWiki_talk:Spam-blacklist
Disallow: /wiki/MediaWiki_talk%3ASpam-blacklist
#
# fi:
# http://bugzilla.wikimedia.org/show_bug.cgi?id=8695
Disallow: /wiki/Wikipedia:Poistettavat_sivut
Disallow: /wiki/K%C3%A4ytt%C3%A4j%C3%A4:
Disallow: /wiki/Käyttäjä:
Disallow: /wiki/Keskustelu_k%C3%A4ytt%C3%A4j%C3%A4st%C3%A4:
Disallow: /wiki/Keskustelu_käyttäjästä:
Disallow: /wiki/Wikipedia:Yll%C3%A4pit%C3%A4j%C3%A4t/
Disallow: /wiki/Wikipedia:Ylläpitäjät/
#
# fr:
Disallow: /wiki/Wikip%C3%A9dia:Pages_%C3%A0_supprimer/
Disallow: /wiki/Wikip%C3%A9dia:Pages_soup%C3%A7onn%C3%A9es_de_viol ation_de_copyright/
#
# he:
Disallow: /wiki/%D7%9E%D7%99%D7%95%D7%97%D7%93:Search
Disallow: /wiki/%D7%9E%D7%99%D7%95%D7%97%D7%93%3ASearch
#
# hu:
Disallow: /wiki/Speci%C3%A1lis:Search
Disallow: /wiki/Speci%C3%A1lis%3ASearch
#
# ja:
Disallow: /wiki/%E7%89%B9%E5%88%A5:Search
Disallow: /wiki/%E7%89%B9%E5%88%A5%3ASearch
#
# ro:
# http://bugzilla.wikimedia.org/show_bug.cgi?id=12546
Disallow: /wiki/Wikipedia:Pagini_de_%C5%9Fters
Disallow: /wiki/Wikipedia%3APagini_de_%C5%9Fters
Disallow: /wiki/Discu%C5%A3ie_Wikipedia:Pagini_de_%C5%9Fters
Disallow: /wiki/Discu%C5%A3ie_Wikipedia%3APagini_de_%C5%9Fters
#
# ru:
Disallow: /wiki/%D0%A1%D0%BF%D0%B5%D1%86%D0%B8%D0%B0%D0%BB%D1%8C%D 0%BD%D1%8B%D0%B5:Search
Disallow: /wiki/%D0%A1%D0%BF%D0%B5%D1%86%D0%B8%D0%B0%D0%BB%D1%8C%D 0%BD%D1%8B%D0%B5%3ASearch
#
# ja:
# https://bugzilla.wikimedia.org/show_bug.cgi?id=5239
Disallow: /wiki/Wikipedia:%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC/
Disallow: /wiki/Wikipedia%3A%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC/
Disallow: /wiki/Wikipedia:%E5%88%A9%E7%94%A8%E8%80%85%E3%83%9A%E3% 83%BC%E3%82%B8%E3%81%AE%E5%89%8A%E9%99%A4%E4%BE%9D %E9%A0%BC
Disallow: /wiki/Wikipedia%3A%E5%88%A9%E7%94%A8%E8%80%85%E3%83%9A%E 3%83%BC%E3%82%B8%E3%81%AE%E5%89%8A%E9%99%A4%E4%BE% 9D%E9%A0%BC
#
# pt:
# https://bugzilla.wikimedia.org/show_bug.cgi?id=5394
Disallow: /wiki/Wikipedia:Páginas_para_eliminar/
Disallow: /wiki/Wikipedia:P%C3%A1ginas_para_eliminar/
Disallow: /wiki/Wikipedia%3AP%C3%A1ginas_para_eliminar/
Disallow: /wiki/Wikipedia_Discussão:Páginas_para_eliminar/
Disallow: /wiki/Wikipedia_Discuss%C3%A3o:P%C3%A1ginas_para_elimina r/
Disallow: /wiki/Wikipedia_Discuss%C3%A3o%3AP%C3%A1ginas_para_elimi nar/
#
# zh:
# https://bugzilla.wikimedia.org/show_bug.cgi?id=5104
Disallow: /wiki/Wikipedia:删除投票/侵权
Disallow: /wiki/Wikipedia:%E5%88%A0%E9%99%A4%E6%8A%95%E7%A5%A8/%E4%BE%B5%E6%9D%83
Disallow: /wiki/Wikipedia:删除投票和请求
Disallow: /wiki/Wikipedia:%E5%88%A0%E9%99%A4%E6%8A%95%E7%A5%A8%E5% 92%8C%E8%AF%B7%E6%B1%82
Disallow: /wiki/Category:快速删除候选
Disallow: /wiki/Category:%E5%BF%AB%E9%80%9F%E5%88%A0%E9%99%A4%E5%8 0%99%E9%80%89
Disallow: /wiki/Category:维基百科需要翻译的文章
Disallow: /wiki/Category:%E7%BB%B4%E5%9F%BA%E7%99%BE%E7%A7%91%E9%9 C%80%E8%A6%81%E7%BF%BB%E8%AF%91%E7%9A%84%E6%96%87% E7%AB%A0
#
# it: - http://bugzilla.wikimedia.org/show_bug.cgi?id=5545
Disallow: /wiki/Wikipedia:Pagine_da_cancellare
Disallow: /wiki/Wikipedia%3APagine_da_cancellare
Disallow: /wiki/Wikipedia:Utenti_problematici
Disallow: /wiki/Wikipedia%3AUtenti_problematici
Disallow: /wiki/Wikipedia:Vandalismi_in_corso
Disallow: /wiki/Wikipedia%3AVandalismi_in_corso
Disallow: /wiki/Wikipedia:Amministratori
Disallow: /wiki/Wikipedia%3AAmministratori
Disallow: /wiki/Wikipedia:Proposte_di_cancellazione_semplificata
Disallow: /wiki/Wikipedia%3AProposte_di_cancellazione_semplificata
Disallow: /wiki/Categoria:Da_cancellare_subito
Disallow: /wiki/Categoria%3ADa_cancellare_subito
Disallow: /wiki/Wikipedia:Sospette_violazioni_di_copyright
Disallow: /wiki/Wikipedia%3ASospette_violazioni_di_copyright
Disallow: /wiki/Categoria:Da_controllare_per_copyright
Disallow: /wiki/Categoria%3ADa_controllare_per_copyright
# added 2007-01-12
Disallow: /wiki/Progetto:Rimozione_contributi_sospetti
Disallow: /wiki/Progetto%3ARimozione_contributi_sospetti
Disallow: /wiki/Categoria:Da_cancellare_subito_per_violazione_inte grale_copyright
Disallow: /wiki/Categoria%3ADa_cancellare_subito_per_violazione_in tegrale_copyright
Disallow: /wiki/Progetto:Cococo
Disallow: /wiki/Progetto%3ACococo
Disallow: /wiki/Discussioni_progetto:Cococo
Disallow: /wiki/Discussioni_progetto%3ACococo
#
# pl.wikipedia.org
# http://bugzilla.wikimedia.org/show_bug.cgi?id=8067
Disallow: /wiki/Wikipedia:Strony_do_usuni%C4%99cia
Disallow: /wiki/Wikipedia%3AStrony_do_usuni%C4%99cia
Disallow: /wiki/Wikipedia:Do_usuni%C4%99cia
Disallow: /wiki/Wikipedia%3ADo_usuni%C4%99cia
Disallow: /wiki/Wikipedia:SDU/
Disallow: /wiki/Wikipedia%3ASDU/
Disallow: /wiki/Wikipedia:Strony_podejrzane_o_naruszenie_praw_auto rskich
Disallow: /wiki/Wikipedia%3AStrony_podejrzane_o_naruszenie_praw_au torskich
#
# en.wikinews:
# https://bugzilla.wikimedia.org/show_bug.cgi?id=5340
Disallow: /wiki/Portal:Prepared_stories/
Disallow: /wiki/Portal%3APrepared_stories/
#
# it.wikinews, http://bugzilla.wikimedia.org/show_bug.cgi?id=9138
Disallow: /wiki/Wikinotizie:Richieste_di_cancellazione
Disallow: /wiki/Wikinotizie:Sospette_violazioni_di_copyright
Disallow: /wiki/Categoria:Da_cancellare_subito
Disallow: /wiki/Categoria:Da_cancellare_subito_per_violazione_inte grale_copyright
Disallow: /wiki/Wikinotizie:Storie_in_preparazione
#
# he.wikipedia, http://bugzilla.wikimedia.org/show_bug.cgi?id=9517
Disallow: /wiki/ויקיפדיה:רשימת_מועמדים_למחיקה/
Disallow: /wiki/ויקיפדיה%3Aרשימת_מועמדים_למחיקה/
Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:% D7%A8%D7%A9%D7%99%D7%9E%D7%AA_%D7%9E%D7%95%D7%A2%D 7%9E%D7%93%D7%99%D7%9D_%D7%9C%D7%9E%D7%97%D7%99%D7 %A7%D7%94/
Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3 A%D7%A8%D7%A9%D7%99%D7%9E%D7%AA_%D7%9E%D7%95%D7%A2 %D7%9E%D7%93%D7%99%D7%9D_%D7%9C%D7%9E%D7%97%D7%99% D7%A7%D7%94/
#
Disallow: /wiki/ויקיפדיה:ערכים_לא_קיימים_ומוגנים
Disallow: /wiki/ויקיפדיה%3Aערכים_לא_קיימים_ומוגנים
Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:% D7%A2%D7%A8%D7%9B%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7% D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D 7%92%D7%A0%D7%99%D7%9D
Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3 A%D7%A2%D7%A8%D7%9B%D7%99%D7%9D_%D7%9C%D7%90_%D7%A 7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95 %D7%92%D7%A0%D7%99%D7%9D
Disallow: /wiki/ויקיפדיה:דפים_לא_קיימים_ומוגנים
Disallow: /wiki/ויקיפדיה%3Aדפים_לא_קיימים_ומוגנים
Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:% D7%93%D7%A4%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99% D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D 7%A0%D7%99%D7%9D
Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3 A%D7%93%D7%A4%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%9 9%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92 %D7%A0%D7%99%D7%9D
#
# sv.wikipedia, #10229
Disallow: /wiki/Wikipedia%3ASidor_f%C3%B6reslagna_f%C3%B6r_raderin g
Disallow: /wiki/Wikipedia:Sidor_f%C3%B6reslagna_f%C3%B6r_radering
Disallow: /wiki/Wikipedia:Sidor_föreslagna_för_radering
#
Disallow: /wiki/Användare
Disallow: /wiki/Anv%C3%A4ndare
#
Disallow: /wiki/Användardiskussion
Disallow: /wiki/Anv%C3%A4ndardiskussion
#
Disallow: /wiki/Wikipedia:Skyddade_sidnamn
Disallow: /wiki/Wikipedia%3ASkyddade_sidnamn
#
## *at least* 1 second please. preferably more :D
## we're disabling this experimentally 11-09-2006
#Crawl-delay: 1
#
Disallow: /wiki/Wikibooks:Votes_for_deletion
#
# 11291
Disallow: /wiki/Wikipedia:Sidor_som_bör_raderas
Disallow: /wiki/Wikipedia:Sidor_som_b%C3%B6r_raderas
Disallow: /wiki/Wikipedia%3ASidor_som_b%C3%B6r_raderas
#
# 11261
Disallow: /wiki/Wikipedia:Requests_for_arbitration/
Disallow: /wiki/Wikipedia%3ARequests_for_arbitration/
Disallow: /wiki/Wikipedia:Requests_for_comment/
Disallow: /wiki/Wikipedia%3ARequests_for_comment/
Disallow: /wiki/Wikipedia:Requests_for_adminship/
Disallow: /wiki/Wikipedia%3ARequests_for_adminship/
#
# 10288
Disallow: /wiki/Wikipedia_talk:Articles_for_deletion/
Disallow: /wiki/Wikipedia_talk%3AArticles_for_deletion/
Disallow: /wiki/Wikipedia_talk:Votes_for_deletion/
Disallow: /wiki/Wikipedia_talk%3AVotes_for_deletion/
Disallow: /wiki/Wikipedia_talk:Pages_for_deletion/
Disallow: /wiki/Wikipedia_talk%3APages_for_deletion/
Disallow: /wiki/Wikipedia_talk:Miscellany_for_deletion/
Disallow: /wiki/Wikipedia_talk%3AMiscellany_for_deletion/
Disallow: /wiki/Wikipedia_talk:Miscellaneous_deletion/
Disallow: /wiki/Wikipedia_talk%3AMiscellaneous_deletion/
#
# 6746
Disallow: /wiki/Wikipedia:Consultas_de_borrado/
Disallow: /wiki/Wikipedia%3AConsultas_de_borrado/
#
# 11432
Disallow: /wiki/Bruker:
Disallow: /wiki/Bruker%3A
Disallow: /wiki/Brukerdiskusjon
Disallow: /wiki/Wikipedia:Administratorer
Disallow: /wiki/Wikipedia%3AAdministratorer
Disallow: /wiki/Wikipedia-diskusjon:Administratorer
Disallow: /wiki/Wikipedia-diskusjon%3AAdministratorer
Disallow: /wiki/Wikipedia:Sletting
Disallow: /wiki/Wikipedia%3ASletting
Disallow: /wiki/Wikipedia-diskusjon:Sletting
Disallow: /wiki/Wikipedia-diskusjon%3ASletting
Disallow: /wiki/Spesial:
Disallow: /wiki/Spesial%3A
#
# working...
Disallow: /wiki/Fundraising_2007/comments
#
# 14793
Disallow: /wiki/Wikipedia:Changing_username
Disallow: /wiki/Wikipedia%3AChanging_username
Disallow: /wiki/Wikipedia:Changing_username/
Disallow: /wiki/Wikipedia%3AChanging_username/
Disallow: /wiki/Wikipedia_talk:Changing_username
Disallow: /wiki/Wikipedia_talk%3AChanging_username
Disallow: /wiki/Wikipedia_talk:Changing_username/
Disallow: /wiki/Wikipedia_talk%3AChanging_username/
#
# 15095
Disallow: /wiki/Wikiquote:Votes_for_deletion/
Disallow: /wiki/Wikiquote%3AVotes_for_deletion/
Disallow: /wiki/Wikiquote_talk:Votes_for_deletion/
Disallow: /wiki/Wikiquote_talk%3AVotes_for_deletion/
Disallow: /wiki/Wikiquote:Votes_for_deletion_archive/
Disallow: /wiki/Wikiquote%3AVotes_for_deletion_archive/
Disallow: /wiki/Wikiquote_talk:Votes_for_deletion_archive/
Disallow: /wiki/Wikiquote_talk%3AVotes_for_deletion_archive/
#
#
#----------------------------------------------------------#
#
#
#
# <syntaxhighlight lang="robots">
#
# Localisable part of robots.txt for en.wikipedia.org
#
# Edit at http://en.wikipedia.org/w/index.php?...xt&action=edit
# Don't add newlines here. All rules set here are active for every user-agent.
#
# Please check any changes using a syntax validator such as http://tool.motoricerca.info/robots-checker.phtml
# Enter http://en.wikipedia.org/robots.txt as the URL to check.
#
# https://bugzilla.wikimedia.org/show_bug.cgi?id=14075
Disallow: /wiki/MediaWiki:Spam-blacklist
Disallow: /wiki/MediaWiki%3ASpam-blacklist
Disallow: /wiki/MediaWiki_talk:Spam-blacklist
Disallow: /wiki/MediaWiki_talk%3ASpam-blacklist
Disallow: /wiki/Wikipedia:WikiProject_Spam
Disallow: /wiki/Wikipedia_talk:WikiProject_Spam
#
# Folks get annoyed when XfD discussions end up the number 1 google hit for
# their name.
# http://bugzilla.wikimedia.org/show_bug.cgi?id=4776
Disallow: /wiki/Wikipedia:Articles_for_deletion/
Disallow: /wiki/Wikipedia%3AArticles_for_deletion/
Disallow: /wiki/Wikipedia:Votes_for_deletion/
Disallow: /wiki/Wikipedia%3AVotes_for_deletion/
Disallow: /wiki/Wikipedia:Pages_for_deletion/
Disallow: /wiki/Wikipedia%3APages_for_deletion/
Disallow: /wiki/Wikipedia:Miscellany_for_deletion/
Disallow: /wiki/Wikipedia%3AMiscellany_for_deletion/
Disallow: /wiki/Wikipedia:Miscellaneous_deletion/
Disallow: /wiki/Wikipedia%3AMiscellaneous_deletion/
Disallow: /wiki/Wikipedia:Categories_for_discussion/
Disallow: /wiki/Wikipedia%3ACategories_for_discussion/
Disallow: /wiki/Wikipedia:Templates_for_deletion/
Disallow: /wiki/Wikipedia%3ATemplates_for_deletion/
Disallow: /wiki/Wikipedia:Redirects_for_discussion/
Disallow: /wiki/Wikipedia%3ARedirects_for_discussion/
Disallow: /wiki/Wikipedia:Deletion_review/
Disallow: /wiki/Wikipedia%3ADeletion_review/
Disallow: /wiki/Wikipedia:WikiProject_Deletion_sorting/
Disallow: /wiki/Wikipedia%3AWikiProject_Deletion_sorting/
Disallow: /wiki/Wikipedia:Templates_for_deletion/
Disallow: /wiki/Wikipedia%3ATemplates_for_deletion/
Disallow: /wiki/Wikipedia:Files_for_deletion/
Disallow: /wiki/Wikipedia%3AFiles_for_deletion/
Disallow: /wiki/Wikipedia:Possibly_unfree_files/
Disallow: /wiki/Wikipedia%3APossibly_unfree_files/
#
# http://bugzilla.wikimedia.org/show_bug.cgi?id=10288
Disallow: /wiki/Wikipedia_talk:Articles_for_deletion/
Disallow: /wiki/Wikipedia_talk%3AArticles_for_deletion/
Disallow: /wiki/Wikipedia_talk:Votes_for_deletion/
Disallow: /wiki/Wikipedia_talk%3AVotes_for_deletion/
Disallow: /wiki/Wikipedia_talk:Pages_for_deletion/
Disallow: /wiki/Wikipedia_talk%3APages_for_deletion/
Disallow: /wiki/Wikipedia_talk:Miscellany_for_deletion/
Disallow: /wiki/Wikipedia_talk%3AMiscellany_for_deletion/
Disallow: /wiki/Wikipedia_talk:Miscellaneous_deletion/
Disallow: /wiki/Wikipedia_talk%3AMiscellaneous_deletion/
Disallow: /wiki/Wikipedia_talk:Templates_for_deletion/
Disallow: /wiki/Wikipedia_talk%3ATemplates_for_deletion/
Disallow: /wiki/Wikipedia_talk:Categories_for_discussion/
Disallow: /wiki/Wikipedia_talk%3ACategories_for_discussion/
Disallow: /wiki/Wikipedia_talk:Deletion_review/
Disallow: /wiki/Wikipedia_talk%3ADeletion_review/
Disallow: /wiki/Wikipedia_talk:WikiProject_Deletion_sorting/
Disallow: /wiki/Wikipedia_talk%3AWikiProject_Deletion_sorting/
Disallow: /wiki/Wikipedia_talk:Templates_for_deletion/
Disallow: /wiki/Wikipedia_talk%3ATemplates_for_deletion/
Disallow: /wiki/Wikipedia_talk:Files_for_deletion/
Disallow: /wiki/Wikipedia_talk%3AFiles_for_deletion/
Disallow: /wiki/Wikipedia_talk:Possibly_unfree_files/
Disallow: /wiki/Wikipedia_talk%3APossibly_unfree_files/
#
Disallow: /wiki/Wikipedia:Copyright_problems
Disallow: /wiki/Wikipedia%3ACopyright_problems
Disallow: /wiki/Wikipedia_talk:Copyright_problems
Disallow: /wiki/Wikipedia_talk%3ACopyright_problems
Disallow: /wiki/Wikipedia:Suspected_copyright_violations
Disallow: /wiki/Wikipedia%3ASuspected_copyright_violations
Disallow: /wiki/Wikipedia_talk:Suspected_copyright_violations
Disallow: /wiki/Wikipedia_talk%3ASuspected_copyright_violations
Disallow: /wiki/Wikipedia:Protected_titles/
Disallow: /wiki/Wikipedia%3AProtected_titles/
Disallow: /wiki/Wikipedia_talk:Protected_titles/
Disallow: /wiki/Wikipedia_talk%3AProtected_titles/
Disallow: /wiki/Wikipedia:Articles_for_creation/
Disallow: /wiki/Wikipedia%3AArticles_for_creation/
Disallow: /wiki/Wikipedia_talk:Articles_for_creation/
Disallow: /wiki/Wikipedia_talk%3AArticles_for_creation/
Disallow: /wiki/Wikipedia_talk:Article_wizard/
Disallow: /wiki/Wikipedia_talk%3AArticle_wizard/
#
# http://bugzilla.wikimedia.org/show_bug.cgi?id=11261
Disallow: /wiki/Wikipedia:Requests_for_arbitration/
Disallow: /wiki/Wikipedia%3ARequests_for_arbitration/
Disallow: /wiki/Wikipedia_talk:Requests_for_arbitration/
Disallow: /wiki/Wikipedia_talk%3ARequests_for_arbitration/
Disallow: /wiki/Wikipedia:Requests_for_comment/
Disallow: /wiki/Wikipedia%3ARequests_for_comment/
Disallow: /wiki/Wikipedia_talk:Requests_for_comment/
Disallow: /wiki/Wikipedia_talk%3ARequests_for_comment/
Disallow: /wiki/Wikipedia:Requests_for_adminship/
Disallow: /wiki/Wikipedia%3ARequests_for_adminship/
Disallow: /wiki/Wikipedia_talk:Requests_for_adminship/
Disallow: /wiki/Wikipedia_talk%3ARequests_for_adminship/
#
# http://bugzilla.wikimedia.org/show_bug.cgi?id=12111
Disallow: /wiki/Wikipedia:Requests_for_checkuser/
Disallow: /wiki/Wikipedia%3ARequests_for_checkuser/
Disallow: /wiki/Wikipedia_talk:Requests_for_checkuser/
Disallow: /wiki/Wikipedia_talk%3ARequests_for_checkuser/
#
# https://bugzilla.wikimedia.org/show_bug.cgi?id=13398
Disallow: /wiki/Wikipedia:WikiProject_Spam/
Disallow: /wiki/Wikipedia%3AWikiProject_Spam/
#
# http://bugzilla.wikimedia.org/show_bug.cgi?id=14793
Disallow: /wiki/Wikipedia:Changing_username
Disallow: /wiki/Wikipedia%3AChanging_username
Disallow: /wiki/Wikipedia:Changing_username/
Disallow: /wiki/Wikipedia%3AChanging_username/
Disallow: /wiki/Wikipedia_talk:Changing_username
Disallow: /wiki/Wikipedia_talk%3AChanging_username
Disallow: /wiki/Wikipedia_talk:Changing_username/
Disallow: /wiki/Wikipedia_talk%3AChanging_username/
#
Disallow: /wiki/Wikipedia:Administrators%27_noticeboard/
Disallow: /wiki/Wikipedia%3AAdministrators%27_noticeboard/
Disallow: /wiki/Wikipedia_talk:Administrators%27_noticeboard/
Disallow: /wiki/Wikipedia_talk%3AAdministrators%27_noticeboard/
Disallow: /wiki/Wikipedia:Community_sanction_noticeboard/
Disallow: /wiki/Wikipedia%3ACommunity_sanction_noticeboard/
Disallow: /wiki/Wikipedia_talk:Community_sanction_noticeboard/
Disallow: /wiki/Wikipedia_talk%3ACommunity_sanction_noticeboard/
Disallow: /wiki/Wikipedia:Bureaucrats%27_noticeboard/
Disallow: /wiki/Wikipedia%3ABureaucrats%27_noticeboard/
Disallow: /wiki/Wikipedia_talk:Bureaucrats%27_noticeboard/
Disallow: /wiki/Wikipedia_talk%3ABureaucrats%27_noticeboard/
#
Disallow: /wiki/Wikipedia:Sockpuppet_investigations/
Disallow: /wiki/Wikipedia%3ASockpuppet_investigations/
Disallow: /wiki/Wikipedia_talk:Sockpuppet_investigations/
Disallow: /wiki/Wikipedia_talk%3ASockpuppet_investigations/
#
Disallow: /wiki/Wikipedia:Neutral_point_of_view/Noticeboard/
Disallow: /wiki/Wikipedia%3ANeutral_point_of_view/Noticeboard/
Disallow: /wiki/Wikipedia_talk:Neutral_point_of_view/Noticeboard/
Disallow: /wiki/Wikipedia_talk%3ANeutral_point_of_view/Noticeboard/
#
Disallow: /wiki/Wikipedia:No_original_research/noticeboard/
Disallow: /wiki/Wikipedia%3ANo_original_research/noticeboard/
Disallow: /wiki/Wikipedia_talk:No_original_research/noticeboard/
Disallow: /wiki/Wikipedia_talk%3ANo_original_research/noticeboard/
#
Disallow: /wiki/Wikipedia:Fringe_theories/Noticeboard/
Disallow: /wiki/Wikipedia%3AFringe_theories/Noticeboard/
Disallow: /wiki/Wikipedia_talk:Fringe_theories/Noticeboard/
Disallow: /wiki/Wikipedia_talk%3AFringe_theories/Noticeboard/
#
Disallow: /wiki/Wikipedia:Conflict_of_interest/Noticeboard/
Disallow: /wiki/Wikipedia%3AConflict_of_interest/Noticeboard/
Disallow: /wiki/Wikipedia_talk:Conflict_of_interest/Noticeboard/
Disallow: /wiki/Wikipedia_talk%3AConflict_of_interest/Noticeboard/
#
Disallow: /wiki/Wikipedia:Long_term_abuse/
Disallow: /wiki/Wikipedia%3ALong_term_abuse/
Disallow: /wiki/Wikipedia_talk:Long_term_abuse/
Disallow: /wiki/Wikipedia_talk%3ALong_term_abuse/
#
Disallow: /wiki/Wikipedia:Abuse_reports/
Disallow: /wiki/Wikipedia%3AAbuse_reports/
Disallow: /wiki/Wikipedia_talk:Abuse_reports/
Disallow: /wiki/Wikipedia_talk%3AAbuse_reports/
#
Disallow: /wiki/Wikipedia:Reliable_sources/Noticeboard/
Disallow: /wiki/Wikipedia%3AReliable_sources/Noticeboard/
Disallow: /wiki/Wikipedia_talk:Reliable_sources/Noticeboard/
Disallow: /wiki/Wikipedia_talk%3AReliable_sources/Noticeboard/
#
Disallow: /wiki/Wikipedia:Suspected_sock_puppets/
Disallow: /wiki/Wikipedia%3ASuspected_sock_puppets/
Disallow: /wiki/Wikipedia_talk:Suspected_sock_puppets/
Disallow: /wiki/Wikipedia_talk%3ASuspected_sock_puppets/
#
Disallow: /wiki/Wikipedia:Biographies_of_living_persons/Noticeboard/
Disallow: /wiki/Wikipedia%3ABiographies_of_living_persons/Noticeboard/
Disallow: /wiki/Wikipedia_talk:Biographies_of_living_persons/Noticeboard/
Disallow: /wiki/Wikipedia_talk%3ABiographies_of_living_persons/Noticeboard/
#
Disallow: /wiki/Wikipedia:Content_noticeboard/
Disallow: /wiki/Wikipedia%3AContent_noticeboard/
Disallow: /wiki/Wikipedia_talk:Content_noticeboard/
Disallow: /wiki/Wikipedia_talk%3AContent_noticeboard/
#
Disallow: /wiki/Template:Editnotices/
Disallow: /wiki/Template%3AEditnotices/
#
Disallow: /wiki/Wikipedia:Arbitration/
Disallow: /wiki/Wikipedia%3AArbitration/
Disallow: /wiki/Wikipedia_talk:Arbitration/
Disallow: /wiki/Wikipedia_talk%3AArbitration/
#
Disallow: /wiki/Wikipedia:Arbitration_Committee/
Disallow: /wiki/Wikipedia%3AArbitration_Committee/
Disallow: /wiki/Wikipedia_talk:Arbitration_Committee/
Disallow: /wiki/Wikipedia_talk%3AArbitration_Committee/
#
Disallow: /wiki/Wikipedia:Arbitration_Committee_Elections
Disallow: /wiki/Wikipedia%3AArbitration_Committee_Elections
Disallow: /wiki/Wikipedia_talk:Arbitration_Committee_Elections
Disallow: /wiki/Wikipedia_talk%3AArbitration_Committee_Elections
#
Disallow: /wiki/Wikipedia:Mediation_Committee/
Disallow: /wiki/Wikipedia%3Mediation_Committee/
Disallow: /wiki/Wikipedia_talk:Mediation_Committee/
Disallow: /wiki/Wikipedia_talk%3Mediation_Committee/
#
Disallow: /wiki/Wikipedia:Mediation_Cabal/Cases/
Disallow: /wiki/Wikipedia%3Mediation_Cabal/Cases/
#
Disallow: /wiki/Wikipedia:Requests_for_bureaucratship/
Disallow: /wiki/Wikipedia%3ARequests_for_bureaucratship/
Disallow: /wiki/Wikipedia_talk:Requests_for_bureaucratship/
Disallow: /wiki/Wikipedia_talk%3ARequests_for_bureaucratship/
#
Disallow: /wiki/Wikipedia:Administrator_review/
Disallow: /wiki/Wikipedia%3AAdministrator_review/
Disallow: /wiki/Wikipedia_talk:Administrator_review/
Disallow: /wiki/Wikipedia_talk%3AAdministrator_review/
#
Disallow: /wiki/Wikipedia:Editor_review/
Disallow: /wiki/Wikipedia%3AEditor_review/
Disallow: /wiki/Wikipedia_talk:Editor_review/
Disallow: /wiki/Wikipedia_talk%3AEditor_review/
#
Disallow: /wiki/Wikipedia:Article_Incubator/
Disallow: /wiki/Wikipedia%3AArticle_Incubator/
Disallow: /wiki/Wikipedia_talk:Article_Incubator/
Disallow: /wiki/Wikipedia_talk%3AArticle_Incubator/
#
Disallow: /wiki/Category:Noindexed_pages
Disallow: /wiki/Category%3ANoindexed_pages
#
# </syntaxhighlight>

Quote:

Originally Posted by Ashimar (Post 618062)

Quote:

Originally Posted by Ashimar (Post 618062)

i c

robots.txt is just a courtesy for bots, but the rules defined dont have to be followed. if its on da web its on da web. but for legit bots like googlebot and archive.org, and not just spammy content scrapers, then ye u outta luck pal

Quote:

Originally Posted by Ashimar (Post 618062)

Interesting

Quote:

Originally Posted by Reikerz (Post 618123)

Essentially the jewish israelites (webmasters) marked their doors (websites) with the blood of a spring lamb (robots.txt) as a signal to the lord (archive.org) to pass over their home (website) and spare it of plague (archival).

Quote:

Originally Posted by Reikerz (Post 618123)

Brilliantly done.