robots.txt tester
Check which of your pages each crawler can reach, before you find out the hard way.
Free, instant, and nothing to sign up for.
This checked one thing. A full audit reads the whole site and tells you what to fix first.
Run the free auditWhat this checks
- Whether each URL is allowed or blocked, for each crawler you name
- Which specific rule made the decision, so you can see what to change
- Any crawler that is blocked from your site entirely
- The sitemap directives your robots.txt declares
Why it matters
robots.txt is the first file a crawler reads and the easiest place on a site to lose traffic without noticing. A single Disallow line in the wrong place can remove a whole section from search, and nothing on the page changes to tell you. It is also where most sites accidentally block AI crawlers, which is now the difference between being quoted in an answer and being invisible in one.
How to read the result
Allowed means the crawler may fetch the page. It does not mean the page will be indexed, and it does not mean the page is any good: robots.txt controls access, not quality. A blocked URL is the more urgent result, because a page a crawler cannot fetch cannot rank at all. Check the rule shown beside each blocked URL before you change anything, since the fix is usually to narrow one line rather than to delete it.
Common questions
Does blocking a page in robots.txt remove it from Google?
Not reliably. Blocking stops Google fetching the page, but a URL that is linked from elsewhere can still appear in results with no description. If you want a page out of search, allow the crawl and use a noindex tag instead, so Google can read the instruction.
Should I block AI crawlers?
That is a business decision, not a technical one. Blocking GPTBot or PerplexityBot keeps your content out of their training and their answers, which also means you cannot be cited by them. Most local businesses want the citation. Check what you are currently blocking before deciding, because it is often unintentional.
Why did my URL come back allowed when Google is not indexing it?
robots.txt is only the first gate. A page can be crawlable and still be excluded by a noindex tag, a canonical pointing elsewhere, thin content or no internal links. This tool answers the access question only.
Other free checks
Each one answers a different question about the same page.