Webmaster
robots.txt tester
Enter a URL to see whether crawlers may fetch it, which line of robots.txt decides, and what’s wrong with the file.
Allowed · Bytespider → /en-US/
No rule matches in the group for *: allowed.
Search and AI crawlers on this path
| Crawler | Who | Result |
|---|---|---|
| Googlebot | Google Search | allowed |
| Bingbot | Bing | allowed |
| DuckDuckBot | DuckDuckGo | allowed |
| Applebot | Apple | allowed |
| GPTBot | OpenAI training | allowed |
| OAI-SearchBot | ChatGPT search | allowed |
| ClaudeBot | Anthropic | allowed |
| Google-Extended | Google AI training | allowed |
| PerplexityBot | Perplexity | allowed |
| CCBot | Common Crawl | allowed |
| Bytespider | ByteDance | allowed |
| Meta-ExternalAgent | Meta AI | allowed |
Sitemaps
The file https://developer.mozilla.org/robots.txt · 0.1 KB
1User-agent: * 2Sitemap: https://developer.mozilla.org/sitemap.xml 3 4Disallow: /api/ 5Disallow: /*/files/ 6Disallow: /media 7