Webmaster
robots.txt tester

Enter a URL to see whether crawlers may fetch it, which line of robots.txt decides, and what’s wrong with the file.
Allowed · Googlebot → /athletic/video/JgevBByl7Ke3iRk/
No rule matches in the group for *, googlebot, googlebot: allowed.
Search and AI crawlers on this path

Crawler	Who	Result
Googlebot	Google Search	allowed
Bingbot	Bing	allowed
DuckDuckBot	DuckDuckGo	allowed
Applebot	Apple	allowed
GPTBot	OpenAI training	blocked
OAI-SearchBot	ChatGPT search	blocked
ClaudeBot	Anthropic	blocked
Google-Extended	Google AI training	blocked
PerplexityBot	Perplexity	blocked
CCBot	Common Crawl	blocked
Bytespider	ByteDance	blocked
Meta-ExternalAgent	Meta AI	blocked
Sitemaps

The file https://www.nytimes.com/robots.txt · 8.3 KB

1# New York Times content is made available for your personal, non-commercial
2# use subject to our Terms of Service here:
3# https://help.nytimes.com/hc/en-us/articles/115014893428-Terms-of-Service.
4# Use of any device, tool, or process designed to data mine or scrape the content
5# using automated means is prohibited without prior written permission from
6# The New York Times Company.  Prohibited uses include but are not limited to:
7# (1) text and data mining activities under Art. 4 of the EU Directive on Copyright in
8# the Digital Single Market;
9# (2) the development of any software, machine learning, artificial intelligence (AI),
10# and/or large language models (LLMs);
11# (3) creating or providing archived or cached data sets containing our content to others; and/or
12# (4) any commercial purposes.
13# Contact https://nytlicensing.com/contact/ for assistance.
14
15User-agent: *
16User-agent: Googlebot
17Disallow: /ads/
18Disallow: /adx/bin/
19Disallow: /athletic/wp/wp-admin/
20Allow: /athletic/wp/wp-admin/admin-ajax.php
21Disallow: /athletic/async-*
22Disallow: /athletic/search/*
23Allow: /athletic/search/$
24Disallow: /athletic/checkout/
25Disallow: /athletic/checkout?plan_id*
26Allow: /athletic/checkout/$
27Disallow: /athletic/checkout2*
28Disallow: /athletic/login/
29Disallow: /athletic/login?login_source*
30Disallow: /athletic/login?ref_page*
31Allow: /athletic/login/$
32Disallow: /athletic/login2/
33Disallow: /athletic/login2?login_source*
34Disallow: /athletic/login2?ref_page*
35Allow: /athletic/login2/$
36Disallow: /athletic/report/
37Disallow: /athletic/*/discuss/*
38Allow: /athletic/live-blogs/discuss/
39Allow: /athletic/nfl/game/discuss/
40Disallow: /athletic/register/
41Disallow: /athletic/register?welcome_redirect*
42Disallow: /athletic/register2/
43Disallow: /athletic/register2?welcome_redirect*
44Disallow: /athletic/betmgm-redirect*
45Disallow: /athletic/cdn-cgi/
46Disallow: /athletic/verizon/*
47Disallow: /athletic/forgot-password/*
48Disallow: /athletic/forgot-password2/*
49Disallow: /athletic/amp-social-login*
50Disallow: /athletic/track-analytics/
51Disallow: /athletic/amp-auth/
52Disallow: /athletic/rss-feed/
53Disallow: /athletic/*?*rss=1
54Disallow: /athletic/global-color-test.php
55Disallow: /athletic/global-font-test.php
56Disallow: /athletic/graphql*
57Disallow: /athletic/api*
58Disallow: /athletic/ip*
59Disallow: /athletic/call-set-cookie-with-context/*
60Disallow: /athletic/get-current-user/
61Disallow: /athletic/pv.json
62Disallow: /athletic/following-feed-test/*
63Disallow: /athletic*/boxscore/*
64Disallow: /athletic/feed-test/
65Disallow: /athletic*/signed-mp3-redirect-url/*
66Disallow: /athletic/embedded-interactive/*
67Disallow: /card/panel/
68Disallow: /panel/
69Disallow: /puzzles/leaderboards/invite/*
70Disallow: /svc
71Allow: /svc/crosswords
72Allow: /svc/games
73Allow: /svc/letter-boxed
74Allow: /svc/spelling-bee
75Allow: /svc/wordle
76Allow: /svc/connections
77Allow: /svc/sudoku
78Allow: /svc/strands
79Allow: /svc/pips
80Disallow: /video/embedded/*
81Disallow: /search
82Disallow: /multiproduct/
83Disallow: /hd/
84Disallow: /inyt/
85Disallow: /*?*query=
86Disallow: /*.pdf$
87Disallow: /*?*login=
88Disallow: /*?*campaignId=
89Disallow: /*?*mcubz=
90Disallow: /*?*smprod=
91Disallow: /*?*ProfileID=
92Disallow: /*?*ListingID=
93Disallow: /*?*campaign_id=
94Disallow: /*?*hybrid=
95Disallow: /*?*entry=
96Disallow: /*?*embed=
97Disallow: /*?ls=
98Disallow: /*?*&ls=
99Disallow: /v1/initialize*
100Disallow: /v1/rgstr*
101Disallow: /wirecutter/wp-admin/
102Disallow: /wirecutter/*.zip$
103Disallow: /wirecutter/*.csv$
104Disallow: /wirecutter/deals/beta
105Disallow: /wirecutter/data-requests
106Disallow: /wirecutter/search
107Disallow: /wirecutter/*?s=
108Disallow: /wirecutter/*&xid=
109Disallow: /wirecutter/*?q=
110Disallow: /wirecutter/*?l=
111Disallow: /wirecutter/*?merchant=
112Disallow: /wirecutter/out/
113Disallow: /search
114Disallow: /subscription/*?*source=
115Disallow: /subscription/*?*onboarded=
116Disallow: /*?*smid=
117Disallow: /*?*partner=
118Disallow: /*?*utm_source=
119Allow: /wirecutter/*?*utm_source=
120Allow: /ads/public/
121Allow: /svc/news/v3/all/pshb.rss
122Allow: /wirecutter/reviews/*?*utm_source=
123Allow: /wirecutter/blog/*?*utm_source=
124Allow: /wirecutter/lists/*?*utm_source=
125Allow: /wirecutter/gifts/*?*utm_source=
126
127
128# Googlebot Specific Rules
129
130User-agent: Googlebot
131Disallow: /athletic*adgroupid*
132Disallow: /athletic*campaignid*
133Disallow: /athletic*ad_id*
134Disallow: /athletic*access_token*
135Disallow: /athletic*amp_reader_id*
136Disallow: /athletic*/?source=*
137Disallow: /athletic/*?*embed=1
138
139
140# Disallow Rules
141
142User-agent: AliyunSecBot
143Disallow: /
144
145User-agent: Amazonbot
146Disallow: /wirecutter/
147
148User-agent: anthropic-ai
149Disallow: /
150
151User-agent: Applebot-Extended
152Disallow: /
153
154User-agent: archive.org_bot
155Disallow: /
156
157User-agent: AudigentAdBot
158Disallow: /
159
160User-agent: AwarioRssBot
161User-agent: AwarioSmartBot
162Disallow: /
163
164User-agent: BLEXBot
165Disallow: /
166
167User-agent: Brightbot
168Disallow: /
169
170User-agent: Bytespider
171Disallow: /
172
173User-agent: CCBot
174Disallow: /
175
176User-agent: ChatGPT-User
177Disallow: /
178
179User-agent: ClaudeBot
180Disallow: /
181
182User-agent: Claude-SearchBot
183Disallow: /
184
185User-agent: Claude-User
186Disallow: /
187
188User-agent: Claude-Web
189Disallow: /
190
191User-agent: cohere-ai
192Disallow: /
193
194User-agent: DataForSeoBot
195Disallow: /
196
197User-agent: Diffbot
198Disallow: /
199
200User-agent: Diffbot-User
201Disallow: /
202
203User-agent: DuckAssistBot
204Disallow: /
205
206User-agent: EchoboxBot
207Disallow: /
208
209User-agent: ExaBot
210Disallow: /
211
212User-agent: ExaSearchBot
213Disallow: /
214
215User-agent: FacebookBot
216Disallow: /
217
218User-agent: FirecrawlAgent
219Disallow: /
220
221User-agent: FriendlyCrawler
222Disallow: /
223
224User-agent: Google-CloudVertexBot
225Disallow: /
226Allow: /wirecutter/
227Allow: /athletic/
228
229User-agent: Google-Extended
230Disallow: /
231
232User-agent: GPTBot
233Disallow: /
234
235User-agent: ImagesiftBot
236Disallow: /
237
238User-agent: Jetslide
239Disallow: /
240
241User-agent: magpie-crawler
242Disallow: /
243
244User-agent: Meta-ExternalAgent
245User-agent: meta-externalagent
246Disallow: /
247
248User-agent: Meta-ExternalFetcher
249User-agent: meta-externalfetcher
250Disallow: /
251
252User-agent: Meta-WebIndexer
253User-agent: meta-webindexer/1.1
254Disallow: /
255
256User-agent: MyCentralAIScraperBot
257Disallow: /
258
259User-agent: NewsNow
260Disallow: /
261
262User-agent: news-please
263Disallow: /
264
265User-agent: OAI-SearchBot
266Disallow: /
267
268User-agent: omgili
269Disallow: /
270
271User-agent: omgilibot
272Disallow: /
273
274User-agent: peer39_crawler
275User-agent: peer39_crawler/1.0
276Disallow: /
277
278User-agent: PerplexityBot
279Disallow: /
280
281User-agent: Perplexity-User
282Disallow: /
283
284User-agent: Poseidon Research Crawler
285Disallow: /
286
287User-agent: quillbot.com
288Disallow: /
289
290User-agent: Quora-Bot
291Disallow: /
292
293User-agent: Scrapy
294Disallow: /
295
296User-agent: SeekrBot
297Disallow: /
298
299User-agent: SeznamHomepageCrawler
300Disallow: /
301
302User-agent: ShapBot
303Disallow: /
304
305User-agent: TaraGroup Intelligent Bot
306Disallow: /
307
308User-agent: Timpibot
309Disallow: /
310
311User-agent: TurnitinBot
312Disallow: /
313
314User-agent: ViennaTinyBot
315Disallow: /
316
317User-agent: YouBot
318Disallow: /
319
320
321# Ad Bot Rules
322
323User-agent: AmazonAdBot
324Allow: /
325
326
327# Social Bot Rules
328
329User-agent: facebookexternalhit
330Allow: /*?*smid=
331
332User-agent: Twitterbot
333Allow: /*?*smid=
334
335User-agent: RedditBot
336Allow: /*?*smid=
337
338# Sitemaps
339
340Sitemap: https://www.nytimes.com/sitemaps/new/news.xml.gz
341Sitemap: https://www.nytimes.com/sitemaps/new/sitemap.xml.gz
342Sitemap: https://www.nytimes.com/sitemaps/new/collections.xml.gz
343Sitemap: https://www.nytimes.com/sitemaps/new/video.xml.gz
344Sitemap: https://www.nytimes.com/sitemaps/new/cooking.xml.gz
345Sitemap: https://www.nytimes.com/sitemaps/new/recipe-collects.xml.gz
346Sitemap: https://www.nytimes.com/sitemaps/new/regions.xml
347Sitemap: https://www.nytimes.com/sitemaps/new/best-sellers.xml
348Sitemap: https://www.nytimes.com/sitemaps/new/subscription-landing-pages.xml
349Sitemap: https://www.nytimes.com/sitemaps/new/weather.xml.gz
350Sitemap: https://www.nytimes.com/sitemaps/new/espanol.xml.gz
351Sitemap: https://www.nytimes.com/sitemaps/new/espanol-collects.xml.gz
352Sitemap: https://www.nytimes.com/wirecutter/sitemapindex.xml
353Sitemap: https://www.nytimes.com/athletic/sitemap-videos-index.xml
354Sitemap: https://www.nytimes.com/athletic/sitemap-authors.xml
355Sitemap: https://www.nytimes.com/athletic/sitemap-verticals.xml
356Sitemap: https://www.nytimes.com/athletic/sitemap-teams.xml
357Sitemap: https://www.nytimes.com/athletic/sitemap-cities.xml
358Sitemap: https://www.nytimes.com/athletic/sitemap-players.xml
359Sitemap: https://www.nytimes.com/athletic/sitemap-tags.xml
360Sitemap: https://www.nytimes.com/athletic/sitemap-stats.xml
361Sitemap: https://www.nytimes.com/athletic/sitemap-schedule.xml
362Sitemap: https://www.nytimes.com/athletic/sitemap-roster.xml
363Sitemap: https://www.nytimes.com/athletic/sitemap.xml
364Sitemap: https://www.nytimes.com/games-assets/v2/assets/sitemap/games.xml
365