robots.txt tester

Search and AI crawlers on this path

Notes

• irobots.txt redirects (301 → 301) to https://en.wikipedia.org/robots.txt. Crawlers follow up to five redirects.
• iCrawl-delay is set. Bing respects it; Google ignores it.

Sitemaps

• https://en.wikipedia.org/w/rest.php/site/v1/sitemap/0

The file https://en.wikipedia.org/robots.txt · 27.6 KB

1# robots.txt for http://www.wikipedia.org/ and friends
2#
3# Please note: There are a lot of pages on this site, and there are
4# some misbehaved spiders out there that go _way_ too fast. If you're
5# irresponsible, your access to the site may be blocked.
6#
7
8# Observed spamming large amounts of https://en.wikipedia.org/?curid=NNNNNN
9# and ignoring 429 ratelimit responses, claims to respect robots:
10# http://mj12bot.com/
11User-agent: MJ12bot
12Disallow: /
13
14# advertising-related bots:
15User-agent: Mediapartners-Google*
16Disallow: /
17
18# Wikipedia work bots:
19User-agent: IsraBot
20Disallow:
21
22User-agent: Orthogaffe
23Disallow:
24
25# Crawlers that are kind enough to obey, but which we'd rather not have
26# unless they're feeding search engines.
27User-agent: UbiCrawler
28Disallow: /
29
30User-agent: DOC
31Disallow: /
32
33User-agent: Zao
34Disallow: /
35
36# Some bots are known to be trouble, particularly those designed to copy
37# entire sites. Please obey robots.txt.
38User-agent: sitecheck.internetseer.com
39Disallow: /
40
41User-agent: Zealbot
42Disallow: /
43
44User-agent: MSIECrawler
45Disallow: /
46
47User-agent: SiteSnagger
48Disallow: /
49
50User-agent: WebStripper
51Disallow: /
52
53User-agent: WebCopier
54Disallow: /
55
56User-agent: Fetch
57Disallow: /
58
59User-agent: Offline Explorer
60Disallow: /
61
62User-agent: Teleport
63Disallow: /
64
65User-agent: TeleportPro
66Disallow: /
67
68User-agent: WebZIP
69Disallow: /
70
71User-agent: linko
72Disallow: /
73
74User-agent: HTTrack
75Disallow: /
76
77User-agent: Microsoft.URL.Control
78Disallow: /
79
80User-agent: Xenu
81Disallow: /
82
83User-agent: larbin
84Disallow: /
85
86User-agent: libwww
87Disallow: /
88
89User-agent: ZyBORG
90Disallow: /
91
92User-agent: Download Ninja
93Disallow: /
94
95# Misbehaving, requests much too fast
96User-agent: fast
97Disallow: /
98
99# Sorry, wget in its recursive mode is a frequent problem.
100# Please read the man page and use it properly; there is a
101# --wait option you can use to set the delay between hits,
102# for instance.
103User-agent: wget
104Disallow: /
105
106# The 'grub' distributed client has been *very* poorly behaved.
107User-agent: grub-client
108Disallow: /
109
110# Doesn't follow robots.txt anyway, but...
111User-agent: k2spider
112Disallow: /
113
114# Hits many times per second, not acceptable
115# http://www.nameprotect.com/botinfo.html
116User-agent: NPBot
117Disallow: /
118
119# A capture bot, downloads gazillions of pages with no public benefit
120# http://www.webreaper.net/
121User-agent: WebReaper
122Disallow: /
123
124# Per their statement, semrushbot respects crawl-delay directives
125# We want them to overall stay within reasonable request rates to
126# the backend (20 rps); keeping in mind that the crawl-delay will
127# be applied by site and not globally by the bot, 5 seconds seem
128# like a reasonable approximation
129User-agent: SemrushBot
130Crawl-delay: 5
131
132#
133# Friendly, low-speed bots are welcome viewing article pages, but not
134# dynamically-generated pages please.
135#
136# Inktomi's "Slurp" can read a minimum delay between hits; if your
137# bot supports such a thing using the 'Crawl-delay' or another
138# instruction, please let us know.
139#
140# There is a special exception for API mobileview to allow dynamic
141# mobile web & app views to load section content.
142# These views aren't HTTP-cached but use parser cache aggressively
143# and don't expose Special-pages and other uncached content.
144#
145# Another exception is for REST API documentation, located at
146# /api/rest_v1/?doc.
147#
148User-agent: *
149Allow: /w/api.php?action=mobileview&
150Allow: /w/load.php?
151Allow: /api/rest_v1/?doc
152Allow: /w/rest.php/site/v1/sitemap
153Disallow: /w/
154Disallow: /api/
155Disallow: /trap/
156Disallow: /wiki/Special:
157Disallow: /wiki/Spezial:
158Disallow: /wiki/Spesial:
159Disallow: /wiki/Special%3A
160Disallow: /wiki/Spezial%3A
161Disallow: /wiki/Spesial%3A
162#
163# ar:
164Disallow: /wiki/%D8%AE%D8%A7%D8%B5:Search
165Disallow: /wiki/%D8%AE%D8%A7%D8%B5%3ASearch
166#
167# dewiki:
168# T6937
169# sensible deletion and meta user discussion pages:
170Disallow: /wiki/Wikipedia:L%C3%B6schkandidaten/
171Disallow: /wiki/Wikipedia:Löschkandidaten/
172Disallow: /wiki/Wikipedia:Vandalensperrung/
173Disallow: /wiki/Wikipedia:Benutzersperrung/
174Disallow: /wiki/Wikipedia:Vermittlungsausschuss/
175Disallow: /wiki/Wikipedia:Administratoren/Probleme/
176Disallow: /wiki/Wikipedia:Adminkandidaturen/
177Disallow: /wiki/Wikipedia:Qualitätssicherung/
178Disallow: /wiki/Wikipedia:Qualit%C3%A4tssicherung/
179# 4937#5
180Disallow: /wiki/Wikipedia:Vandalismusmeldung/
181Disallow: /wiki/Wikipedia:Gesperrte_Lemmata/
182Disallow: /wiki/Wikipedia:Löschprüfung/
183Disallow: /wiki/Wikipedia:L%C3%B6schprüfung/
184Disallow: /wiki/Wikipedia:Administratoren/Notizen/
185Disallow: /wiki/Wikipedia:Schiedsgericht/Anfragen/
186Disallow: /wiki/Wikipedia:L%C3%B6schpr%C3%BCfung/
187# T14111
188Disallow: /wiki/Wikipedia:Checkuser/
189Disallow: /wiki/Wikipedia_Diskussion:Checkuser/
190Disallow: /wiki/Wikipedia_Diskussion:Adminkandidaturen/
191# T15961
192Disallow: /wiki/Wikipedia:Spam-Blacklist-Log
193Disallow: /wiki/Wikipedia%3ASpam-Blacklist-Log
194Disallow: /wiki/Wikipedia_Diskussion:Spam-Blacklist-Log
195Disallow: /wiki/Wikipedia_Diskussion%3ASpam-Blacklist-Log
196#
197# enwiki:
198# Folks get annoyed when VfD discussions end up the number 1 google hit for
199# their name. See T6776
200Disallow: /wiki/Wikipedia:Articles_for_deletion/
201Disallow: /wiki/Wikipedia%3AArticles_for_deletion/
202Disallow: /wiki/Wikipedia:Votes_for_deletion/
203Disallow: /wiki/Wikipedia%3AVotes_for_deletion/
204Disallow: /wiki/Wikipedia:Pages_for_deletion/
205Disallow: /wiki/Wikipedia%3APages_for_deletion/
206Disallow: /wiki/Wikipedia:Miscellany_for_deletion/
207Disallow: /wiki/Wikipedia%3AMiscellany_for_deletion/
208Disallow: /wiki/Wikipedia:Miscellaneous_deletion/
209Disallow: /wiki/Wikipedia%3AMiscellaneous_deletion/
210Disallow: /wiki/Wikipedia:Copyright_problems
211Disallow: /wiki/Wikipedia%3ACopyright_problems
212Disallow: /wiki/Wikipedia:Protected_titles/
213Disallow: /wiki/Wikipedia%3AProtected_titles/
214# T15398
215Disallow: /wiki/Wikipedia:WikiProject_Spam/
216Disallow: /wiki/Wikipedia%3AWikiProject_Spam/
217# T16075
218Disallow: /wiki/MediaWiki:Spam-blacklist
219Disallow: /wiki/MediaWiki%3ASpam-blacklist
220Disallow: /wiki/MediaWiki_talk:Spam-blacklist
221Disallow: /wiki/MediaWiki_talk%3ASpam-blacklist
222# T13261
223Disallow: /wiki/Wikipedia:Requests_for_arbitration/
224Disallow: /wiki/Wikipedia%3ARequests_for_arbitration/
225Disallow: /wiki/Wikipedia:Requests_for_comment/
226Disallow: /wiki/Wikipedia%3ARequests_for_comment/
227Disallow: /wiki/Wikipedia:Requests_for_adminship/
228Disallow: /wiki/Wikipedia%3ARequests_for_adminship/
229# T12288
230Disallow: /wiki/Wikipedia_talk:Articles_for_deletion/
231Disallow: /wiki/Wikipedia_talk%3AArticles_for_deletion/
232Disallow: /wiki/Wikipedia_talk:Votes_for_deletion/
233Disallow: /wiki/Wikipedia_talk%3AVotes_for_deletion/
234Disallow: /wiki/Wikipedia_talk:Pages_for_deletion/
235Disallow: /wiki/Wikipedia_talk%3APages_for_deletion/
236Disallow: /wiki/Wikipedia_talk:Miscellany_for_deletion/
237Disallow: /wiki/Wikipedia_talk%3AMiscellany_for_deletion/
238Disallow: /wiki/Wikipedia_talk:Miscellaneous_deletion/
239Disallow: /wiki/Wikipedia_talk%3AMiscellaneous_deletion/
240# T16793
241Disallow: /wiki/Wikipedia:Changing_username
242Disallow: /wiki/Wikipedia%3AChanging_username
243Disallow: /wiki/Wikipedia:Changing_username/
244Disallow: /wiki/Wikipedia%3AChanging_username/
245Disallow: /wiki/Wikipedia_talk:Changing_username
246Disallow: /wiki/Wikipedia_talk%3AChanging_username
247Disallow: /wiki/Wikipedia_talk:Changing_username/
248Disallow: /wiki/Wikipedia_talk%3AChanging_username/
249#
250# eswiki:
251# T8746
252Disallow: /wiki/Wikipedia:Consultas_de_borrado/
253Disallow: /wiki/Wikipedia%3AConsultas_de_borrado/
254#
255# fiwiki:
256# T10695
257Disallow: /wiki/Wikipedia:Poistettavat_sivut
258Disallow: /wiki/K%C3%A4ytt%C3%A4j%C3%A4:
259Disallow: /wiki/Käyttäjä:
260Disallow: /wiki/Keskustelu_k%C3%A4ytt%C3%A4j%C3%A4st%C3%A4:
261Disallow: /wiki/Keskustelu_käyttäjästä:
262Disallow: /wiki/Wikipedia:Yll%C3%A4pit%C3%A4j%C3%A4t/
263Disallow: /wiki/Wikipedia:Ylläpitäjät/
264#
265# hewiki:
266Disallow: /wiki/%D7%9E%D7%99%D7%95%D7%97%D7%93:Search
267Disallow: /wiki/%D7%9E%D7%99%D7%95%D7%97%D7%93%3ASearch
268#T11517
269Disallow: /wiki/ויקיפדיה:רשימת_מועמדים_למחיקה/
270Disallow: /wiki/ויקיפדיה%3Aרשימת_מועמדים_למחיקה/
271Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:%D7%A8%D7%A9%D7%99%D7%9E%D7%AA_%D7%9E%D7%95%D7%A2%D7%9E%D7%93%D7%99%D7%9D_%D7%9C%D7%9E%D7%97%D7%99%D7%A7%D7%94/
272Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3A%D7%A8%D7%A9%D7%99%D7%9E%D7%AA_%D7%9E%D7%95%D7%A2%D7%9E%D7%93%D7%99%D7%9D_%D7%9C%D7%9E%D7%97%D7%99%D7%A7%D7%94/
273Disallow: /wiki/ויקיפדיה:ערכים_לא_קיימים_ומוגנים
274Disallow: /wiki/ויקיפדיה%3Aערכים_לא_קיימים_ומוגנים
275Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:%D7%A2%D7%A8%D7%9B%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D
276Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3A%D7%A2%D7%A8%D7%9B%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D
277Disallow: /wiki/ויקיפדיה:דפים_לא_קיימים_ומוגנים
278Disallow: /wiki/ויקיפדיה%3Aדפים_לא_קיימים_ומוגנים
279Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:%D7%93%D7%A4%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D
280Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3A%D7%93%D7%A4%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D
281#
282# huwiki:
283Disallow: /wiki/Speci%C3%A1lis:Search
284Disallow: /wiki/Speci%C3%A1lis%3ASearch
285#
286# itwiki:
287# T7545
288Disallow: /wiki/Wikipedia:Pagine_da_cancellare
289Disallow: /wiki/Wikipedia%3APagine_da_cancellare
290Disallow: /wiki/Wikipedia:Utenti_problematici
291Disallow: /wiki/Wikipedia%3AUtenti_problematici
292Disallow: /wiki/Wikipedia:Vandalismi_in_corso
293Disallow: /wiki/Wikipedia%3AVandalismi_in_corso
294Disallow: /wiki/Wikipedia:Amministratori
295Disallow: /wiki/Wikipedia%3AAmministratori
296Disallow: /wiki/Wikipedia:Proposte_di_cancellazione_semplificata
297Disallow: /wiki/Wikipedia%3AProposte_di_cancellazione_semplificata
298Disallow: /wiki/Categoria:Da_cancellare_subito
299Disallow: /wiki/Categoria%3ADa_cancellare_subito
300Disallow: /wiki/Wikipedia:Sospette_violazioni_di_copyright
301Disallow: /wiki/Wikipedia%3ASospette_violazioni_di_copyright
302Disallow: /wiki/Categoria:Da_controllare_per_copyright
303Disallow: /wiki/Categoria%3ADa_controllare_per_copyright
304Disallow: /wiki/Progetto:Rimozione_contributi_sospetti
305Disallow: /wiki/Progetto%3ARimozione_contributi_sospetti
306Disallow: /wiki/Categoria:Da_cancellare_subito_per_violazione_integrale_copyright
307Disallow: /wiki/Categoria%3ADa_cancellare_subito_per_violazione_integrale_copyright
308Disallow: /wiki/Progetto:Cococo
309Disallow: /wiki/Progetto%3ACococo
310Disallow: /wiki/Discussioni_progetto:Cococo
311Disallow: /wiki/Discussioni_progetto%3ACococo
312#
313# jawiki
314Disallow: /wiki/%E7%89%B9%E5%88%A5:Search
315Disallow: /wiki/%E7%89%B9%E5%88%A5%3ASearch
316# T7239
317Disallow: /wiki/Wikipedia:%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC/
318Disallow: /wiki/Wikipedia%3A%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC/
319Disallow: /wiki/Wikipedia:%E5%88%A9%E7%94%A8%E8%80%85%E3%83%9A%E3%83%BC%E3%82%B8%E3%81%AE%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC
320Disallow: /wiki/Wikipedia%3A%E5%88%A9%E7%94%A8%E8%80%85%E3%83%9A%E3%83%BC%E3%82%B8%E3%81%AE%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC
321# nowiki
322# T13432
323Disallow: /wiki/Bruker:
324Disallow: /wiki/Bruker%3A
325Disallow: /wiki/Brukerdiskusjon
326Disallow: /wiki/Wikipedia:Administratorer
327Disallow: /wiki/Wikipedia%3AAdministratorer
328Disallow: /wiki/Wikipedia-diskusjon:Administratorer
329Disallow: /wiki/Wikipedia-diskusjon%3AAdministratorer
330Disallow: /wiki/Wikipedia:Sletting
331Disallow: /wiki/Wikipedia%3ASletting
332Disallow: /wiki/Wikipedia-diskusjon:Sletting
333Disallow: /wiki/Wikipedia-diskusjon%3ASletting
334#
335# plwiki
336# T10067
337Disallow: /wiki/Wikipedia:Strony_do_usuni%C4%99cia
338Disallow: /wiki/Wikipedia%3AStrony_do_usuni%C4%99cia
339Disallow: /wiki/Wikipedia:Do_usuni%C4%99cia
340Disallow: /wiki/Wikipedia%3ADo_usuni%C4%99cia
341Disallow: /wiki/Wikipedia:SDU/
342Disallow: /wiki/Wikipedia%3ASDU/
343Disallow: /wiki/Wikipedia:Strony_podejrzane_o_naruszenie_praw_autorskich
344Disallow: /wiki/Wikipedia%3AStrony_podejrzane_o_naruszenie_praw_autorskich
345#
346# ptwiki:
347# T7394
348Disallow: /wiki/Wikipedia:Páginas_para_eliminar/
349Disallow: /wiki/Wikipedia:P%C3%A1ginas_para_eliminar/
350Disallow: /wiki/Wikipedia%3AP%C3%A1ginas_para_eliminar/
351Disallow: /wiki/Wikipedia_Discussão:Páginas_para_eliminar/
352Disallow: /wiki/Wikipedia_Discuss%C3%A3o:P%C3%A1ginas_para_eliminar/
353Disallow: /wiki/Wikipedia_Discuss%C3%A3o%3AP%C3%A1ginas_para_eliminar/
354#
355# rowiki:
356# T14546
357Disallow: /wiki/Wikipedia:Pagini_de_%C5%9Fters
358Disallow: /wiki/Wikipedia%3APagini_de_%C5%9Fters
359Disallow: /wiki/Discu%C5%A3ie_Wikipedia:Pagini_de_%C5%9Fters
360Disallow: /wiki/Discu%C5%A3ie_Wikipedia%3APagini_de_%C5%9Fters
361#
362# ruwiki:
363Disallow: /wiki/%D0%A1%D0%BF%D0%B5%D1%86%D0%B8%D0%B0%D0%BB%D1%8C%D0%BD%D1%8B%D0%B5:Search
364Disallow: /wiki/%D0%A1%D0%BF%D0%B5%D1%86%D0%B8%D0%B0%D0%BB%D1%8C%D0%BD%D1%8B%D0%B5%3ASearch
365#
366# svwiki:
367# T12229
368Disallow: /wiki/Wikipedia%3ASidor_f%C3%B6reslagna_f%C3%B6r_radering
369Disallow: /wiki/Wikipedia:Sidor_f%C3%B6reslagna_f%C3%B6r_radering
370Disallow: /wiki/Wikipedia:Sidor_föreslagna_för_radering
371Disallow: /wiki/Användare
372Disallow: /wiki/Anv%C3%A4ndare
373Disallow: /wiki/Användardiskussion
374Disallow: /wiki/Anv%C3%A4ndardiskussion
375Disallow: /wiki/Wikipedia:Skyddade_sidnamn
376Disallow: /wiki/Wikipedia%3ASkyddade_sidnamn
377# T13291
378Disallow: /wiki/Wikipedia:Sidor_som_bör_raderas
379Disallow: /wiki/Wikipedia:Sidor_som_b%C3%B6r_raderas
380Disallow: /wiki/Wikipedia%3ASidor_som_b%C3%B6r_raderas
381#
382# zhwiki:
383# T7104
384Disallow: /wiki/Wikipedia:删除投票/侵权
385Disallow: /wiki/Wikipedia:%E5%88%A0%E9%99%A4%E6%8A%95%E7%A5%A8/%E4%BE%B5%E6%9D%83
386Disallow: /wiki/Wikipedia:删除投票和请求
387Disallow: /wiki/Wikipedia:%E5%88%A0%E9%99%A4%E6%8A%95%E7%A5%A8%E5%92%8C%E8%AF%B7%E6%B1%82
388Disallow: /wiki/Category:快速删除候选
389Disallow: /wiki/Category:%E5%BF%AB%E9%80%9F%E5%88%A0%E9%99%A4%E5%80%99%E9%80%89
390Disallow: /wiki/Category:维基百科需要翻译的文章
391Disallow: /wiki/Category:%E7%BB%B4%E5%9F%BA%E7%99%BE%E7%A7%91%E9%9C%80%E8%A6%81%E7%BF%BB%E8%AF%91%E7%9A%84%E6%96%87%E7%AB%A0
392#
393# sister projects
394#
395# enwikinews:
396# T7340
397Disallow: /wiki/Portal:Prepared_stories/
398Disallow: /wiki/Portal%3APrepared_stories/
399#
400# itwikinews