NexaYori Tools Help
Help / robots.txt tester

robots.txt tester

Reads a site’s robots.txt and tells you whether a crawler (Googlebot, Bing, GPTBot, ClaudeBot…) may fetch a given URL, which line decides it, and what’s wrong with the file.

Open robots.txt tester →

How to use it

  1. Enter a full URL (the path matters) or just a domain to test the home page.
  2. Pick a crawler, or leave it on Googlebot.
  3. The top box gives the verdict and the exact rule; the table shows how every major search and AI crawler is treated on that path.
  4. The file is shown below with the deciding line highlighted.

Reading the results

User-agent group
A block of rules for one or more crawlers. A crawler uses its own group if there is one, otherwise the * group.
Allow / Disallow
Rules that permit or block paths starting with the given text.
Longest match wins
When several rules match, the most specific (longest) one decides. A tie goes to Allow.
* and $
* matches anything; $ at the end means “the URL ends here”. Disallow: /*.pdf$ blocks every PDF.
Crawl-delay
Seconds between requests. Bing honors it; Google ignores it.

Examples

From the command line

curl "https://tools.nexayori.com/robots?q=github.com%2Fsettings"
curl "https://tools.nexayori.com/robots?q=github.com%2Fsettings&format=json"
qURL to test (domain = home page)
uaCrawler name (default Googlebot)
formattext (default for curl), json, or html

Questions

Does robots.txt keep a page out of Google?

No: it stops crawling, not indexing. A blocked page can still appear (without a description) if other sites link to it. Use a noindex tag, and don’t block the page, to keep it out.

What if there’s no robots.txt?

Everything may be crawled. That’s fine; add one mainly to list your sitemap.

Do AI crawlers obey robots.txt?

The major ones listed here say they do. Google-Extended controls Google’s AI training without affecting search.

Limitations