|
|
#1
04-11-2012, 11:33 PM
|
|||
|
what the hell is Robots.txt?
I heard the Wayback Machine is good to look up old websites that arent around anymore and were archived. Well apparently you cant view some because of "Robots.txt" ...
Trying to learn about it the page just confuses me. i just want to see the goddamn webpage. Anyone know a way around it? what the hell is this shit? the link was EQ Easter eggs over at http://wayback.archive.org/web/*/htt...asterquest.htm
__________________
 | ||
|
|
|||
|
#2
04-11-2012, 11:34 PM
|
|||
|
do you even read the text under the error, you gigantic fucktard?
http://en.wikipedia.org/wiki/Robots_exclusion_standard | ||
|
|
|||
|
#3
04-11-2012, 11:41 PM
|
|||
|
#
# robots.txt for http://www.wikipedia.org/ and friends # # Please note: There are a lot of pages on this site, and there are # some misbehaved spiders out there that go _way_ too fast. If you're # irresponsible, your access to the site may be blocked. # # advertising-related bots: User-agent: Mediapartners-Google* Disallow: / # Wikipedia work bots: User-agent: IsraBot Disallow: User-agent: Orthogaffe Disallow: # Crawlers that are kind enough to obey, but which we'd rather not have # unless they're feeding search engines. User-agent: UbiCrawler Disallow: / User-agent: DOC Disallow: / User-agent: Zao Disallow: / # Some bots are known to be trouble, particularly those designed to copy # entire sites. Please obey robots.txt. User-agent: sitecheck.internetseer.com Disallow: / User-agent: Zealbot Disallow: / User-agent: MSIECrawler Disallow: / User-agent: SiteSnagger Disallow: / User-agent: WebStripper Disallow: / User-agent: WebCopier Disallow: / User-agent: Fetch Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebZIP Disallow: / User-agent: linko Disallow: / User-agent: HTTrack Disallow: / User-agent: Microsoft.URL.Control Disallow: / User-agent: Xenu Disallow: / User-agent: larbin Disallow: / User-agent: libwww Disallow: / User-agent: ZyBORG Disallow: / User-agent: Download Ninja Disallow: / # # Sorry, wget in its recursive mode is a frequent problem. # Please read the man page and use it properly; there is a # --wait option you can use to set the delay between hits, # for instance. # User-agent: wget Disallow: / # # The 'grub' distributed client has been *very* poorly behaved. # User-agent: grub-client Disallow: / # # Doesn't follow robots.txt anyway, but... # User-agent: k2spider Disallow: / # # Hits many times per second, not acceptable # http://www.nameprotect.com/botinfo.html User-agent: NPBot Disallow: / # A capture bot, downloads gazillions of pages with no public benefit # http://www.webreaper.net/ User-agent: WebReaper Disallow: / # Don't allow the wayback-maschine to index user-pages #User-agent: ia_archiver #Disallow: /wiki/User #Disallow: /wiki/Benutzer # # Friendly, low-speed bots are welcome viewing article pages, but not # dynamically-generated pages please. # # Inktomi's "Slurp" can read a minimum delay between hits; if your # bot supports such a thing using the 'Crawl-delay' or another # instruction, please let us know. # User-agent: * Disallow: /w/ Disallow: /trap/ Disallow: /wiki/Especial:Search Disallow: /wiki/Especial%3ASearch Disallow: /wiki/Special:Collection Disallow: /wiki/Spezial:Sammlung Disallow: /wiki/Special:Random Disallow: /wiki/Special%3ARandom Disallow: /wiki/Special:Search Disallow: /wiki/Special%3ASearch Disallow: /wiki/Spesial:Search Disallow: /wiki/Spesial%3ASearch Disallow: /wiki/Spezial:Search Disallow: /wiki/Spezial%3ASearch Disallow: /wiki/Specjalna:Search Disallow: /wiki/Specjalna%3ASearch Disallow: /wiki/Speciaal:Search Disallow: /wiki/Speciaal%3ASearch Disallow: /wiki/Speciaal:Random Disallow: /wiki/Speciaal%3ARandom Disallow: /wiki/Speciel:Search Disallow: /wiki/Speciel%3ASearch Disallow: /wiki/Speciale:Search Disallow: /wiki/Speciale%3ASearch Disallow: /wiki/Istimewa:Search Disallow: /wiki/Istimewa%3ASearch Disallow: /wiki/Toiminnot:Search Disallow: /wiki/Toiminnot%3ASearch # # ar: Disallow: /wiki/%D8%AE%D8%A7%D8%B5:Search Disallow: /wiki/%D8%AE%D8%A7%D8%B5%3ASearch # # de: # http://bugzilla.wikimedia.org/show_bug.cgi?id=4937 # sensible deletion and meta user discussion pages: Disallow: /wiki/Wikipedia:L%C3%B6schkandidaten/ Disallow: /wiki/Wikipedia:Löschkandidaten/ Disallow: /wiki/Wikipedia:Vandalensperrung/ Disallow: /wiki/Wikipedia:Benutzersperrung/ Disallow: /wiki/Wikipedia:Vermittlungsausschuss/ Disallow: /wiki/Wikipedia:Administratoren/Probleme/ Disallow: /wiki/Wikipedia:Adminkandidaturen/ Disallow: /wiki/Wikipedia:Qualitätssicherung/ Disallow: /wiki/Wikipedia:Qualit%C3%A4tssicherung/ # Search- and random-page Disallow: /wiki/Spezial:Suche Disallow: /wiki/Special:Suche Disallow: /wiki/Spezial:Zufällige_Seite Disallow: /wiki/Spezial:Zuf%C3%A4llige_Seite Disallow: /wiki/Special:Zufällige_Seite Disallow: /wiki/Special:Zuf%C3%A4llige_Seite # 4937#5 Disallow: /wiki/Wikipedia:Vandalismusmeldung/ Disallow: /wiki/Wikipedia:Gesperrte_Lemmata/ Disallow: /wiki/Wikipedia:Löschprüfung/ Disallow: /wiki/Wikipedia:L%C3%B6schprüfung/ Disallow: /wiki/Wikipedia:Administratoren/Notizen/ Disallow: /wiki/Wikipedia:Schiedsgericht/Anfragen/ Disallow: /wiki/Wikipedia:L%C3%B6schpr%C3%BCfung/ # http://bugzilla.wikimedia.org/show_bug.cgi?id=12111 Disallow: /wiki/Wikipedia:Checkuser/ Disallow: /wiki/Wikipedia_Diskussion:Checkuser/ Disallow: /wiki/Wikipedia_Diskussion:Adminkandidaturen/ # https://bugzilla.wikimedia.org/show_bug.cgi?id=13961 Disallow: /wiki/Wikipedia:Spam-Blacklist-Log Disallow: /wiki/Wikipedia%3ASpam-Blacklist-Log Disallow: /wiki/Wikipedia_Diskussion:Spam-Blacklist-Log Disallow: /wiki/Wikipedia_Diskussion%3ASpam-Blacklist-Log # # Folks get annoyed when VfD discussions end up the number 1 google hit for # their name. See bugzilla bug #4776 # en: Disallow: /wiki/Wikipedia:Articles_for_deletion/ Disallow: /wiki/Wikipedia%3AArticles_for_deletion/ Disallow: /wiki/Wikipedia:Votes_for_deletion/ Disallow: /wiki/Wikipedia%3AVotes_for_deletion/ Disallow: /wiki/Wikipedia:Pages_for_deletion/ Disallow: /wiki/Wikipedia%3APages_for_deletion/ Disallow: /wiki/Wikipedia:Miscellany_for_deletion/ Disallow: /wiki/Wikipedia%3AMiscellany_for_deletion/ Disallow: /wiki/Wikipedia:Miscellaneous_deletion/ Disallow: /wiki/Wikipedia%3AMiscellaneous_deletion/ Disallow: /wiki/Wikipedia:Copyright_problems Disallow: /wiki/Wikipedia%3ACopyright_problems Disallow: /wiki/Wikipedia:Protected_titles/ Disallow: /wiki/Wikipedia%3AProtected_titles/ # https://bugzilla.wikimedia.org/show_bug.cgi?id=13398 Disallow: /wiki/Wikipedia:WikiProject_Spam/ Disallow: /wiki/Wikipedia%3AWikiProject_Spam/ # https://bugzilla.wikimedia.org/show_bug.cgi?id=14075 Disallow: /wiki/MediaWiki:Spam-blacklist Disallow: /wiki/MediaWiki%3ASpam-blacklist Disallow: /wiki/MediaWiki_talk:Spam-blacklist Disallow: /wiki/MediaWiki_talk%3ASpam-blacklist # # fi: # http://bugzilla.wikimedia.org/show_bug.cgi?id=8695 Disallow: /wiki/Wikipedia:Poistettavat_sivut Disallow: /wiki/K%C3%A4ytt%C3%A4j%C3%A4: Disallow: /wiki/Käyttäjä: Disallow: /wiki/Keskustelu_k%C3%A4ytt%C3%A4j%C3%A4st%C3%A4: Disallow: /wiki/Keskustelu_käyttäjästä: Disallow: /wiki/Wikipedia:Yll%C3%A4pit%C3%A4j%C3%A4t/ Disallow: /wiki/Wikipedia:Ylläpitäjät/ # # fr: Disallow: /wiki/Wikip%C3%A9dia:Pages_%C3%A0_supprimer/ Disallow: /wiki/Wikip%C3%A9dia:Pages_soup%C3%A7onn%C3%A9es_de_viol ation_de_copyright/ # # he: Disallow: /wiki/%D7%9E%D7%99%D7%95%D7%97%D7%93:Search Disallow: /wiki/%D7%9E%D7%99%D7%95%D7%97%D7%93%3ASearch # # hu: Disallow: /wiki/Speci%C3%A1lis:Search Disallow: /wiki/Speci%C3%A1lis%3ASearch # # ja: Disallow: /wiki/%E7%89%B9%E5%88%A5:Search Disallow: /wiki/%E7%89%B9%E5%88%A5%3ASearch # # ro: # http://bugzilla.wikimedia.org/show_bug.cgi?id=12546 Disallow: /wiki/Wikipedia:Pagini_de_%C5%9Fters Disallow: /wiki/Wikipedia%3APagini_de_%C5%9Fters Disallow: /wiki/Discu%C5%A3ie_Wikipedia:Pagini_de_%C5%9Fters Disallow: /wiki/Discu%C5%A3ie_Wikipedia%3APagini_de_%C5%9Fters # # ru: Disallow: /wiki/%D0%A1%D0%BF%D0%B5%D1%86%D0%B8%D0%B0%D0%BB%D1%8C%D 0%BD%D1%8B%D0%B5:Search Disallow: /wiki/%D0%A1%D0%BF%D0%B5%D1%86%D0%B8%D0%B0%D0%BB%D1%8C%D 0%BD%D1%8B%D0%B5%3ASearch # # ja: # https://bugzilla.wikimedia.org/show_bug.cgi?id=5239 Disallow: /wiki/Wikipedia:%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC/ Disallow: /wiki/Wikipedia%3A%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC/ Disallow: /wiki/Wikipedia:%E5%88%A9%E7%94%A8%E8%80%85%E3%83%9A%E3% 83%BC%E3%82%B8%E3%81%AE%E5%89%8A%E9%99%A4%E4%BE%9D %E9%A0%BC Disallow: /wiki/Wikipedia%3A%E5%88%A9%E7%94%A8%E8%80%85%E3%83%9A%E 3%83%BC%E3%82%B8%E3%81%AE%E5%89%8A%E9%99%A4%E4%BE% 9D%E9%A0%BC # # pt: # https://bugzilla.wikimedia.org/show_bug.cgi?id=5394 Disallow: /wiki/Wikipedia:Páginas_para_eliminar/ Disallow: /wiki/Wikipedia:P%C3%A1ginas_para_eliminar/ Disallow: /wiki/Wikipedia%3AP%C3%A1ginas_para_eliminar/ Disallow: /wiki/Wikipedia_Discussão:Páginas_para_eliminar/ Disallow: /wiki/Wikipedia_Discuss%C3%A3o:P%C3%A1ginas_para_elimina r/ Disallow: /wiki/Wikipedia_Discuss%C3%A3o%3AP%C3%A1ginas_para_elimi nar/ # # zh: # https://bugzilla.wikimedia.org/show_bug.cgi?id=5104 Disallow: /wiki/Wikipedia:删除投票/侵权 Disallow: /wiki/Wikipedia:%E5%88%A0%E9%99%A4%E6%8A%95%E7%A5%A8/%E4%BE%B5%E6%9D%83 Disallow: /wiki/Wikipedia:删除投票和请求 Disallow: /wiki/Wikipedia:%E5%88%A0%E9%99%A4%E6%8A%95%E7%A5%A8%E5% 92%8C%E8%AF%B7%E6%B1%82 Disallow: /wiki/Category:快速删除候选 Disallow: /wiki/Category:%E5%BF%AB%E9%80%9F%E5%88%A0%E9%99%A4%E5%8 0%99%E9%80%89 Disallow: /wiki/Category:维基百科需要翻译的文章 Disallow: /wiki/Category:%E7%BB%B4%E5%9F%BA%E7%99%BE%E7%A7%91%E9%9 C%80%E8%A6%81%E7%BF%BB%E8%AF%91%E7%9A%84%E6%96%87% E7%AB%A0 # # it: - http://bugzilla.wikimedia.org/show_bug.cgi?id=5545 Disallow: /wiki/Wikipedia:Pagine_da_cancellare Disallow: /wiki/Wikipedia%3APagine_da_cancellare Disallow: /wiki/Wikipedia:Utenti_problematici Disallow: /wiki/Wikipedia%3AUtenti_problematici Disallow: /wiki/Wikipedia:Vandalismi_in_corso Disallow: /wiki/Wikipedia%3AVandalismi_in_corso Disallow: /wiki/Wikipedia:Amministratori Disallow: /wiki/Wikipedia%3AAmministratori Disallow: /wiki/Wikipedia:Proposte_di_cancellazione_semplificata Disallow: /wiki/Wikipedia%3AProposte_di_cancellazione_semplificata Disallow: /wiki/Categoria[You must be logged in to view images. Log in or Register.]a_cancellare_subito Disallow: /wiki/Categoria%3ADa_cancellare_subito Disallow: /wiki/Wikipedia:Sospette_violazioni_di_copyright Disallow: /wiki/Wikipedia%3ASospette_violazioni_di_copyright Disallow: /wiki/Categoria[You must be logged in to view images. Log in or Register.]a_controllare_per_copyright Disallow: /wiki/Categoria%3ADa_controllare_per_copyright # added 2007-01-12 Disallow: /wiki/Progetto:Rimozione_contributi_sospetti Disallow: /wiki/Progetto%3ARimozione_contributi_sospetti Disallow: /wiki/Categoria[You must be logged in to view images. Log in or Register.]a_cancellare_subito_per_violazione_inte grale_copyright Disallow: /wiki/Categoria%3ADa_cancellare_subito_per_violazione_in tegrale_copyright Disallow: /wiki/Progetto:Cococo Disallow: /wiki/Progetto%3ACococo Disallow: /wiki/Discussioni_progetto:Cococo Disallow: /wiki/Discussioni_progetto%3ACococo # # pl.wikipedia.org # http://bugzilla.wikimedia.org/show_bug.cgi?id=8067 Disallow: /wiki/Wikipedia:Strony_do_usuni%C4%99cia Disallow: /wiki/Wikipedia%3AStrony_do_usuni%C4%99cia Disallow: /wiki/Wikipedia[You must be logged in to view images. Log in or Register.]o_usuni%C4%99cia Disallow: /wiki/Wikipedia%3ADo_usuni%C4%99cia Disallow: /wiki/Wikipedia:SDU/ Disallow: /wiki/Wikipedia%3ASDU/ Disallow: /wiki/Wikipedia:Strony_podejrzane_o_naruszenie_praw_auto rskich Disallow: /wiki/Wikipedia%3AStrony_podejrzane_o_naruszenie_praw_au torskich # # en.wikinews: # https://bugzilla.wikimedia.org/show_bug.cgi?id=5340 Disallow: /wiki/Portal:Prepared_stories/ Disallow: /wiki/Portal%3APrepared_stories/ # # it.wikinews, http://bugzilla.wikimedia.org/show_bug.cgi?id=9138 Disallow: /wiki/Wikinotizie:Richieste_di_cancellazione Disallow: /wiki/Wikinotizie:Sospette_violazioni_di_copyright Disallow: /wiki/Categoria[You must be logged in to view images. Log in or Register.]a_cancellare_subito Disallow: /wiki/Categoria[You must be logged in to view images. Log in or Register.]a_cancellare_subito_per_violazione_inte grale_copyright Disallow: /wiki/Wikinotizie:Storie_in_preparazione # # he.wikipedia, http://bugzilla.wikimedia.org/show_bug.cgi?id=9517 Disallow: /wiki/ויקיפדיה:רשימת_מועמדים_למחיקה/ Disallow: /wiki/ויקיפדיה%3Aרשימת_מועמדים_למחיקה/ Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:% D7%A8%D7%A9%D7%99%D7%9E%D7%AA_%D7%9E%D7%95%D7%A2%D 7%9E%D7%93%D7%99%D7%9D_%D7%9C%D7%9E%D7%97%D7%99%D7 %A7%D7%94/ Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3 A%D7%A8%D7%A9%D7%99%D7%9E%D7%AA_%D7%9E%D7%95%D7%A2 %D7%9E%D7%93%D7%99%D7%9D_%D7%9C%D7%9E%D7%97%D7%99% D7%A7%D7%94/ # Disallow: /wiki/ויקיפדיה:ערכים_לא_קיימים_ומוגנים Disallow: /wiki/ויקיפדיה%3Aערכים_לא_קיימים_ומוגנים Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:% D7%A2%D7%A8%D7%9B%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7% D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D 7%92%D7%A0%D7%99%D7%9D Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3 A%D7%A2%D7%A8%D7%9B%D7%99%D7%9D_%D7%9C%D7%90_%D7%A 7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95 %D7%92%D7%A0%D7%99%D7%9D Disallow: /wiki/ויקיפדיה:דפים_לא_קיימים_ומוגנים Disallow: /wiki/ויקיפדיה%3Aדפים_לא_קיימים_ומוגנים Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:% D7%93%D7%A4%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99% D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D 7%A0%D7%99%D7%9D Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3 A%D7%93%D7%A4%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%9 9%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92 %D7%A0%D7%99%D7%9D # # sv.wikipedia, #10229 Disallow: /wiki/Wikipedia%3ASidor_f%C3%B6reslagna_f%C3%B6r_raderin g Disallow: /wiki/Wikipedia:Sidor_f%C3%B6reslagna_f%C3%B6r_radering Disallow: /wiki/Wikipedia:Sidor_föreslagna_för_radering # Disallow: /wiki/Användare Disallow: /wiki/Anv%C3%A4ndare # Disallow: /wiki/Användardiskussion Disallow: /wiki/Anv%C3%A4ndardiskussion # Disallow: /wiki/Wikipedia:Skyddade_sidnamn Disallow: /wiki/Wikipedia%3ASkyddade_sidnamn # ## *at least* 1 second please. preferably more [You must be logged in to view images. Log in or Register.] ## we're disabling this experimentally 11-09-2006 #Crawl-delay: 1 # Disallow: /wiki/Wikibooks:Votes_for_deletion # # 11291 Disallow: /wiki/Wikipedia:Sidor_som_bör_raderas Disallow: /wiki/Wikipedia:Sidor_som_b%C3%B6r_raderas Disallow: /wiki/Wikipedia%3ASidor_som_b%C3%B6r_raderas # # 11261 Disallow: /wiki/Wikipedia:Requests_for_arbitration/ Disallow: /wiki/Wikipedia%3ARequests_for_arbitration/ Disallow: /wiki/Wikipedia:Requests_for_comment/ Disallow: /wiki/Wikipedia%3ARequests_for_comment/ Disallow: /wiki/Wikipedia:Requests_for_adminship/ Disallow: /wiki/Wikipedia%3ARequests_for_adminship/ # # 10288 Disallow: /wiki/Wikipedia_talk:Articles_for_deletion/ Disallow: /wiki/Wikipedia_talk%3AArticles_for_deletion/ Disallow: /wiki/Wikipedia_talk:Votes_for_deletion/ Disallow: /wiki/Wikipedia_talk%3AVotes_for_deletion/ Disallow: /wiki/Wikipedia_talk:Pages_for_deletion/ Disallow: /wiki/Wikipedia_talk%3APages_for_deletion/ Disallow: /wiki/Wikipedia_talk:Miscellany_for_deletion/ Disallow: /wiki/Wikipedia_talk%3AMiscellany_for_deletion/ Disallow: /wiki/Wikipedia_talk:Miscellaneous_deletion/ Disallow: /wiki/Wikipedia_talk%3AMiscellaneous_deletion/ # # 6746 Disallow: /wiki/Wikipedia:Consultas_de_borrado/ Disallow: /wiki/Wikipedia%3AConsultas_de_borrado/ # # 11432 Disallow: /wiki/Bruker: Disallow: /wiki/Bruker%3A Disallow: /wiki/Brukerdiskusjon Disallow: /wiki/Wikipedia:Administratorer Disallow: /wiki/Wikipedia%3AAdministratorer Disallow: /wiki/Wikipedia-diskusjon:Administratorer Disallow: /wiki/Wikipedia-diskusjon%3AAdministratorer Disallow: /wiki/Wikipedia:Sletting Disallow: /wiki/Wikipedia%3ASletting Disallow: /wiki/Wikipedia-diskusjon:Sletting Disallow: /wiki/Wikipedia-diskusjon%3ASletting Disallow: /wiki/Spesial: Disallow: /wiki/Spesial%3A # # working... Disallow: /wiki/Fundraising_2007/comments # # 14793 Disallow: /wiki/Wikipedia:Changing_username Disallow: /wiki/Wikipedia%3AChanging_username Disallow: /wiki/Wikipedia:Changing_username/ Disallow: /wiki/Wikipedia%3AChanging_username/ Disallow: /wiki/Wikipedia_talk:Changing_username Disallow: /wiki/Wikipedia_talk%3AChanging_username Disallow: /wiki/Wikipedia_talk:Changing_username/ Disallow: /wiki/Wikipedia_talk%3AChanging_username/ # # 15095 Disallow: /wiki/Wikiquote:Votes_for_deletion/ Disallow: /wiki/Wikiquote%3AVotes_for_deletion/ Disallow: /wiki/Wikiquote_talk:Votes_for_deletion/ Disallow: /wiki/Wikiquote_talk%3AVotes_for_deletion/ Disallow: /wiki/Wikiquote:Votes_for_deletion_archive/ Disallow: /wiki/Wikiquote%3AVotes_for_deletion_archive/ Disallow: /wiki/Wikiquote_talk:Votes_for_deletion_archive/ Disallow: /wiki/Wikiquote_talk%3AVotes_for_deletion_archive/ # # #----------------------------------------------------------# # # # # <!-- Please do not remove the space at the start of this line, it breaks the rendering. http://www.robotstxt.org/orig.html says spaces before comments are OK. --><syntaxhighlight lang="robots"> # # Localisable part of robots.txt for en.wikipedia.org # # Edit at http://en.wikipedia.org/w/index.php?...xt&action=edit # Don't add newlines here. All rules set here are active for every user-agent. # # Please check any changes using a syntax validator such as http://tool.motoricerca.info/robots-checker.phtml # Enter http://en.wikipedia.org/robots.txt as the URL to check. # # https://bugzilla.wikimedia.org/show_bug.cgi?id=14075 Disallow: /wiki/MediaWiki:Spam-blacklist Disallow: /wiki/MediaWiki%3ASpam-blacklist Disallow: /wiki/MediaWiki_talk:Spam-blacklist Disallow: /wiki/MediaWiki_talk%3ASpam-blacklist Disallow: /wiki/Wikipedia:WikiProject_Spam Disallow: /wiki/Wikipedia_talk:WikiProject_Spam # # Folks get annoyed when XfD discussions end up the number 1 google hit for # their name. # http://bugzilla.wikimedia.org/show_bug.cgi?id=4776 Disallow: /wiki/Wikipedia:Articles_for_deletion/ Disallow: /wiki/Wikipedia%3AArticles_for_deletion/ Disallow: /wiki/Wikipedia:Votes_for_deletion/ Disallow: /wiki/Wikipedia%3AVotes_for_deletion/ Disallow: /wiki/Wikipedia:Pages_for_deletion/ Disallow: /wiki/Wikipedia%3APages_for_deletion/ Disallow: /wiki/Wikipedia:Miscellany_for_deletion/ Disallow: /wiki/Wikipedia%3AMiscellany_for_deletion/ Disallow: /wiki/Wikipedia:Miscellaneous_deletion/ Disallow: /wiki/Wikipedia%3AMiscellaneous_deletion/ Disallow: /wiki/Wikipedia:Categories_for_discussion/ Disallow: /wiki/Wikipedia%3ACategories_for_discussion/ Disallow: /wiki/Wikipedia:Templates_for_deletion/ Disallow: /wiki/Wikipedia%3ATemplates_for_deletion/ Disallow: /wiki/Wikipedia:Redirects_for_discussion/ Disallow: /wiki/Wikipedia%3ARedirects_for_discussion/ Disallow: /wiki/Wikipedia[You must be logged in to view images. Log in or Register.]eletion_review/ Disallow: /wiki/Wikipedia%3ADeletion_review/ Disallow: /wiki/Wikipedia:WikiProject_Deletion_sorting/ Disallow: /wiki/Wikipedia%3AWikiProject_Deletion_sorting/ Disallow: /wiki/Wikipedia:Templates_for_deletion/ Disallow: /wiki/Wikipedia%3ATemplates_for_deletion/ Disallow: /wiki/Wikipedia:Files_for_deletion/ Disallow: /wiki/Wikipedia%3AFiles_for_deletion/ Disallow: /wiki/Wikipedia:Possibly_unfree_files/ Disallow: /wiki/Wikipedia%3APossibly_unfree_files/ # # http://bugzilla.wikimedia.org/show_bug.cgi?id=10288 Disallow: /wiki/Wikipedia_talk:Articles_for_deletion/ Disallow: /wiki/Wikipedia_talk%3AArticles_for_deletion/ Disallow: /wiki/Wikipedia_talk:Votes_for_deletion/ Disallow: /wiki/Wikipedia_talk%3AVotes_for_deletion/ Disallow: /wiki/Wikipedia_talk:Pages_for_deletion/ Disallow: /wiki/Wikipedia_talk%3APages_for_deletion/ Disallow: /wiki/Wikipedia_talk:Miscellany_for_deletion/ Disallow: /wiki/Wikipedia_talk%3AMiscellany_for_deletion/ Disallow: /wiki/Wikipedia_talk:Miscellaneous_deletion/ Disallow: /wiki/Wikipedia_talk%3AMiscellaneous_deletion/ Disallow: /wiki/Wikipedia_talk:Templates_for_deletion/ Disallow: /wiki/Wikipedia_talk%3ATemplates_for_deletion/ Disallow: /wiki/Wikipedia_talk:Categories_for_discussion/ Disallow: /wiki/Wikipedia_talk%3ACategories_for_discussion/ Disallow: /wiki/Wikipedia_talk[You must be logged in to view images. Log in or Register.]eletion_review/ Disallow: /wiki/Wikipedia_talk%3ADeletion_review/ Disallow: /wiki/Wikipedia_talk:WikiProject_Deletion_sorting/ Disallow: /wiki/Wikipedia_talk%3AWikiProject_Deletion_sorting/ Disallow: /wiki/Wikipedia_talk:Templates_for_deletion/ Disallow: /wiki/Wikipedia_talk%3ATemplates_for_deletion/ Disallow: /wiki/Wikipedia_talk:Files_for_deletion/ Disallow: /wiki/Wikipedia_talk%3AFiles_for_deletion/ Disallow: /wiki/Wikipedia_talk:Possibly_unfree_files/ Disallow: /wiki/Wikipedia_talk%3APossibly_unfree_files/ # Disallow: /wiki/Wikipedia:Copyright_problems Disallow: /wiki/Wikipedia%3ACopyright_problems Disallow: /wiki/Wikipedia_talk:Copyright_problems Disallow: /wiki/Wikipedia_talk%3ACopyright_problems Disallow: /wiki/Wikipedia:Suspected_copyright_violations Disallow: /wiki/Wikipedia%3ASuspected_copyright_violations Disallow: /wiki/Wikipedia_talk:Suspected_copyright_violations Disallow: /wiki/Wikipedia_talk%3ASuspected_copyright_violations Disallow: /wiki/Wikipedia:Protected_titles/ Disallow: /wiki/Wikipedia%3AProtected_titles/ Disallow: /wiki/Wikipedia_talk:Protected_titles/ Disallow: /wiki/Wikipedia_talk%3AProtected_titles/ Disallow: /wiki/Wikipedia:Articles_for_creation/ Disallow: /wiki/Wikipedia%3AArticles_for_creation/ Disallow: /wiki/Wikipedia_talk:Articles_for_creation/ Disallow: /wiki/Wikipedia_talk%3AArticles_for_creation/ Disallow: /wiki/Wikipedia_talk:Article_wizard/ Disallow: /wiki/Wikipedia_talk%3AArticle_wizard/ # # http://bugzilla.wikimedia.org/show_bug.cgi?id=11261 Disallow: /wiki/Wikipedia:Requests_for_arbitration/ Disallow: /wiki/Wikipedia%3ARequests_for_arbitration/ Disallow: /wiki/Wikipedia_talk:Requests_for_arbitration/ Disallow: /wiki/Wikipedia_talk%3ARequests_for_arbitration/ Disallow: /wiki/Wikipedia:Requests_for_comment/ Disallow: /wiki/Wikipedia%3ARequests_for_comment/ Disallow: /wiki/Wikipedia_talk:Requests_for_comment/ Disallow: /wiki/Wikipedia_talk%3ARequests_for_comment/ Disallow: /wiki/Wikipedia:Requests_for_adminship/ Disallow: /wiki/Wikipedia%3ARequests_for_adminship/ Disallow: /wiki/Wikipedia_talk:Requests_for_adminship/ Disallow: /wiki/Wikipedia_talk%3ARequests_for_adminship/ # # http://bugzilla.wikimedia.org/show_bug.cgi?id=12111 Disallow: /wiki/Wikipedia:Requests_for_checkuser/ Disallow: /wiki/Wikipedia%3ARequests_for_checkuser/ Disallow: /wiki/Wikipedia_talk:Requests_for_checkuser/ Disallow: /wiki/Wikipedia_talk%3ARequests_for_checkuser/ # # https://bugzilla.wikimedia.org/show_bug.cgi?id=13398 Disallow: /wiki/Wikipedia:WikiProject_Spam/ Disallow: /wiki/Wikipedia%3AWikiProject_Spam/ # # http://bugzilla.wikimedia.org/show_bug.cgi?id=14793 Disallow: /wiki/Wikipedia:Changing_username Disallow: /wiki/Wikipedia%3AChanging_username Disallow: /wiki/Wikipedia:Changing_username/ Disallow: /wiki/Wikipedia%3AChanging_username/ Disallow: /wiki/Wikipedia_talk:Changing_username Disallow: /wiki/Wikipedia_talk%3AChanging_username Disallow: /wiki/Wikipedia_talk:Changing_username/ Disallow: /wiki/Wikipedia_talk%3AChanging_username/ # Disallow: /wiki/Wikipedia:Administrators%27_noticeboard/ Disallow: /wiki/Wikipedia%3AAdministrators%27_noticeboard/ Disallow: /wiki/Wikipedia_talk:Administrators%27_noticeboard/ Disallow: /wiki/Wikipedia_talk%3AAdministrators%27_noticeboard/ Disallow: /wiki/Wikipedia:Community_sanction_noticeboard/ Disallow: /wiki/Wikipedia%3ACommunity_sanction_noticeboard/ Disallow: /wiki/Wikipedia_talk:Community_sanction_noticeboard/ Disallow: /wiki/Wikipedia_talk%3ACommunity_sanction_noticeboard/ Disallow: /wiki/Wikipedia:Bureaucrats%27_noticeboard/ Disallow: /wiki/Wikipedia%3ABureaucrats%27_noticeboard/ Disallow: /wiki/Wikipedia_talk:Bureaucrats%27_noticeboard/ Disallow: /wiki/Wikipedia_talk%3ABureaucrats%27_noticeboard/ # Disallow: /wiki/Wikipedia:Sockpuppet_investigations/ Disallow: /wiki/Wikipedia%3ASockpuppet_investigations/ Disallow: /wiki/Wikipedia_talk:Sockpuppet_investigations/ Disallow: /wiki/Wikipedia_talk%3ASockpuppet_investigations/ # Disallow: /wiki/Wikipedia:Neutral_point_of_view/Noticeboard/ Disallow: /wiki/Wikipedia%3ANeutral_point_of_view/Noticeboard/ Disallow: /wiki/Wikipedia_talk:Neutral_point_of_view/Noticeboard/ Disallow: /wiki/Wikipedia_talk%3ANeutral_point_of_view/Noticeboard/ # Disallow: /wiki/Wikipedia:No_original_research/noticeboard/ Disallow: /wiki/Wikipedia%3ANo_original_research/noticeboard/ Disallow: /wiki/Wikipedia_talk:No_original_research/noticeboard/ Disallow: /wiki/Wikipedia_talk%3ANo_original_research/noticeboard/ # Disallow: /wiki/Wikipedia:Fringe_theories/Noticeboard/ Disallow: /wiki/Wikipedia%3AFringe_theories/Noticeboard/ Disallow: /wiki/Wikipedia_talk:Fringe_theories/Noticeboard/ Disallow: /wiki/Wikipedia_talk%3AFringe_theories/Noticeboard/ # Disallow: /wiki/Wikipedia:Conflict_of_interest/Noticeboard/ Disallow: /wiki/Wikipedia%3AConflict_of_interest/Noticeboard/ Disallow: /wiki/Wikipedia_talk:Conflict_of_interest/Noticeboard/ Disallow: /wiki/Wikipedia_talk%3AConflict_of_interest/Noticeboard/ # Disallow: /wiki/Wikipedia:Long_term_abuse/ Disallow: /wiki/Wikipedia%3ALong_term_abuse/ Disallow: /wiki/Wikipedia_talk:Long_term_abuse/ Disallow: /wiki/Wikipedia_talk%3ALong_term_abuse/ # Disallow: /wiki/Wikipedia:Abuse_reports/ Disallow: /wiki/Wikipedia%3AAbuse_reports/ Disallow: /wiki/Wikipedia_talk:Abuse_reports/ Disallow: /wiki/Wikipedia_talk%3AAbuse_reports/ # Disallow: /wiki/Wikipedia:Reliable_sources/Noticeboard/ Disallow: /wiki/Wikipedia%3AReliable_sources/Noticeboard/ Disallow: /wiki/Wikipedia_talk:Reliable_sources/Noticeboard/ Disallow: /wiki/Wikipedia_talk%3AReliable_sources/Noticeboard/ # Disallow: /wiki/Wikipedia:Suspected_sock_puppets/ Disallow: /wiki/Wikipedia%3ASuspected_sock_puppets/ Disallow: /wiki/Wikipedia_talk:Suspected_sock_puppets/ Disallow: /wiki/Wikipedia_talk%3ASuspected_sock_puppets/ # Disallow: /wiki/Wikipedia:Biographies_of_living_persons/Noticeboard/ Disallow: /wiki/Wikipedia%3ABiographies_of_living_persons/Noticeboard/ Disallow: /wiki/Wikipedia_talk:Biographies_of_living_persons/Noticeboard/ Disallow: /wiki/Wikipedia_talk%3ABiographies_of_living_persons/Noticeboard/ # Disallow: /wiki/Wikipedia:Content_noticeboard/ Disallow: /wiki/Wikipedia%3AContent_noticeboard/ Disallow: /wiki/Wikipedia_talk:Content_noticeboard/ Disallow: /wiki/Wikipedia_talk%3AContent_noticeboard/ # Disallow: /wiki/Template:Editnotices/ Disallow: /wiki/Template%3AEditnotices/ # Disallow: /wiki/Wikipedia:Arbitration/ Disallow: /wiki/Wikipedia%3AArbitration/ Disallow: /wiki/Wikipedia_talk:Arbitration/ Disallow: /wiki/Wikipedia_talk%3AArbitration/ # Disallow: /wiki/Wikipedia:Arbitration_Committee/ Disallow: /wiki/Wikipedia%3AArbitration_Committee/ Disallow: /wiki/Wikipedia_talk:Arbitration_Committee/ Disallow: /wiki/Wikipedia_talk%3AArbitration_Committee/ # Disallow: /wiki/Wikipedia:Arbitration_Committee_Elections Disallow: /wiki/Wikipedia%3AArbitration_Committee_Elections Disallow: /wiki/Wikipedia_talk:Arbitration_Committee_Elections Disallow: /wiki/Wikipedia_talk%3AArbitration_Committee_Elections # Disallow: /wiki/Wikipedia:Mediation_Committee/ Disallow: /wiki/Wikipedia%3Mediation_Committee/ Disallow: /wiki/Wikipedia_talk:Mediation_Committee/ Disallow: /wiki/Wikipedia_talk%3Mediation_Committee/ # Disallow: /wiki/Wikipedia:Mediation_Cabal/Cases/ Disallow: /wiki/Wikipedia%3Mediation_Cabal/Cases/ # Disallow: /wiki/Wikipedia:Requests_for_bureaucratship/ Disallow: /wiki/Wikipedia%3ARequests_for_bureaucratship/ Disallow: /wiki/Wikipedia_talk:Requests_for_bureaucratship/ Disallow: /wiki/Wikipedia_talk%3ARequests_for_bureaucratship/ # Disallow: /wiki/Wikipedia:Administrator_review/ Disallow: /wiki/Wikipedia%3AAdministrator_review/ Disallow: /wiki/Wikipedia_talk:Administrator_review/ Disallow: /wiki/Wikipedia_talk%3AAdministrator_review/ # Disallow: /wiki/Wikipedia:Editor_review/ Disallow: /wiki/Wikipedia%3AEditor_review/ Disallow: /wiki/Wikipedia_talk:Editor_review/ Disallow: /wiki/Wikipedia_talk%3AEditor_review/ # Disallow: /wiki/Wikipedia:Article_Incubator/ Disallow: /wiki/Wikipedia%3AArticle_Incubator/ Disallow: /wiki/Wikipedia_talk:Article_Incubator/ Disallow: /wiki/Wikipedia_talk%3AArticle_Incubator/ # Disallow: /wiki/Category:Noindexed_pages Disallow: /wiki/Category%3ANoindexed_pages # # </syntaxhighlight>
__________________
| ||
|
|
|||
|
#5
04-11-2012, 11:50 PM
|
||||
|
Quote:
| |||
|
|
||||
|
#6
04-11-2012, 11:51 PM
|
||||
|
Quote:
__________________
 | |||
|
|
||||
|
#7
04-11-2012, 11:52 PM
|
|||
|
robots.txt is just a courtesy for bots, but the rules defined dont have to be followed. if its on da web its on da web. but for legit bots like googlebot and archive.org, and not just spammy content scrapers, then ye u outta luck pal
| ||
|
|
|||
|
#8
04-12-2012, 12:57 AM
|
||||
|
Quote:
| |||
|
|
||||
|
#9
04-12-2012, 12:59 AM
|
||||
|
Quote:
| |||
|
|
||||
|
#10
04-12-2012, 04:53 PM
|
||||
|
Quote:
__________________
Transience - 60 Erudite Enchanter
| |||
|
|
||||
 |
|
|
Linear Mode
