About this tool
The AI Crawler Accessibility Checker shows whether 16 documented AI crawlers and control tokens (including GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, and Applebot-Extended) are allowed to fetch a specific page. For each one it names the exact robots.txt line that decides the result, separates AI search crawlers from training crawlers, and tests whether your server or firewall answers differently when a request carries an AI user agent. You can also paste a draft robots.txt and test it locally before you deploy it.
How to use
- Enter a page or site URL and select Check crawlers, or switch to Paste robots.txt to test a draft file against any path.
- Read the three summary tiles: AI search, user-triggered, and AI training crawlers allowed for that path.
- Check each crawler row for the deciding rule, its line number, and which group (its own or *) it came from.
- Look for “Server blocked this user agent” warnings, which point to a firewall, CDN, or bot-management rule rather than robots.txt.
- Use Starter robots.txt rules to copy a policy, merge it into your file, and run the check again.
Training, AI search, and user-triggered crawlers are different decisions
Most AI companies now run separate agents for separate jobs. OpenAI uses GPTBot for model training, OAI-SearchBot to index pages for ChatGPT search, and ChatGPT-User when a person asks ChatGPT to open a page. Anthropic splits ClaudeBot, Claude-SearchBot, and Claude-User the same way, and Perplexity separates PerplexityBot from Perplexity-User. Blocking a training crawler opts your content out of future training data; blocking a search crawler can stop AI answers from citing and linking your page. The checker groups results by purpose so you can see which of those choices your robots.txt is actually making.
How the robots.txt rules are evaluated
The parser follows RFC 9309, the Robots Exclusion Protocol standard. User-agent lines are matched case-insensitively against each crawler’s product token, and every group naming that token is merged. If no group names it, the * group applies. Within the chosen group, the longest matching Allow or Disallow path wins, and an Allow beats a Disallow of the same length. Paths are matched against the URL path plus its query string, with * matching any characters and $ anchoring the end. A 4xx response for robots.txt means crawlers may fetch everything, while a 5xx response or no response means compliant crawlers must assume everything is disallowed.
Control tokens that never visit your site
Google-Extended and Applebot-Extended are robots.txt tokens, not crawlers. Google fetches pages with its normal Googlebot user agents and reads Google-Extended only to decide whether that content may be used for Gemini training and grounding; Google states it does not affect inclusion or ranking in Google Search. Applebot-Extended works the same way for Apple’s foundation models. Because they send no requests, the server test is not run for them, and blocking them never removes a page from search results.
What the server user-agent test shows
Robots.txt is only a request. Many sites also block AI bots at a CDN, firewall, or bot-management layer by user agent. For the crawlers that fetch pages, the checker sends one request with a user agent containing that crawler’s token and compares the status code with a normal browser request. A 401, 403, 406, 429, 451, or 503 for the crawler while the browser gets a success suggests a user-agent rule. The requests come from our server, not from the operator’s published IP ranges, so a real crawler can be treated differently, for example, verified-bot rules may allow it while blocking imitations. Googlebot, Bingbot, and Applebot are not tested because sites commonly reject unverified copies of them.
Robots meta tags and X-Robots-Tag still apply
Crawler access is only the first step. The checker also reads the page’s robots meta tags and X-Robots-Tag header. A noindex directive keeps the page out of search indexes, and nosnippet or max-snippet:0 limits how much text can be shown, which Google says also applies to AI Overviews and AI Mode. Some sites add noai or noimageai; these are not part of any standard, and none of the crawler operators listed here document support for them.
Example: block training but stay citable
A common policy is to disallow GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, and meta-externalagent with Disallow: /, while leaving a User-agent: * group that allows everything else. With that file, the checker should show every training crawler as Blocked and every AI search crawler as Allowed. Because user-triggered agents such as ChatGPT-User and Perplexity-User may not follow robots.txt at all, use authentication or server-side rules if content must not be fetched on a user’s behalf.
Continue this workflow
Use the adjacent tool when the next step calls for a different input, output, or method.
- GEO Score CheckerScore the same page on content, structure, and evidence signals that affect AI search citations.
- Opengraph CheckerCheck the title, description, and share-preview tags on the page you just tested.
- URL ParserBreak a URL into path and query to see exactly what robots.txt rules match against.
Assumptions and limitations
- Only the 16 documented crawlers and tokens listed are evaluated. Other AI agents, including undocumented ones, are not covered.
- Robots.txt is voluntary. Crawlers that ignore it, and user-triggered fetchers whose operators say robots.txt may not apply, can still request blocked pages.
- Product-token matching is exact. Some operators, notably Google, apply extra fallback rules (for example, Googlebot-Image falling back to a Googlebot group) that this checker does not model.
- Percent-encoded and unencoded forms of the same path are compared as typed, so rules that differ only in encoding may be evaluated differently from a particular crawler.
- The server test sends requests from our server with an imitation user agent. Firewalls that verify crawler IP addresses may treat the real crawler differently, so treat that column as a signal, not proof.
- Only the first 500 KiB of robots.txt and the first 1 MB of the page are read. Pages behind logins, geoblocking, or JavaScript challenges may not be checkable.
- Requests to non-default ports and to hostnames that resolve to private or internal network addresses are blocked.
Sources and standards
These authoritative references were used to verify the method and guidance on this page.
- RFC 9309: Robots Exclusion Protocol — IETF
- Overview of OpenAI Crawlers — OpenAI
- Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic
- Google’s common crawlers (Google-Extended) — Google Search Central
- AI features and your website — Google Search Central
- Perplexity Crawlers — Perplexity
- About Applebot — Apple
- Meta Web Crawlers — Meta for Developers
- CCBot — Common Crawl
Frequently asked questions
How do I check if GPTBot is blocked on my site?
Enter your page URL and look at the GPTBot row. It shows Allowed or Blocked for that exact path, plus the robots.txt line and group that decided it. The server test underneath shows whether your firewall or CDN rejects requests that carry the GPTBot user agent.
If I block GPTBot, will my site disappear from ChatGPT search?
No. GPTBot is OpenAI’s training crawler. ChatGPT search uses OAI-SearchBot, which has its own robots.txt token, so you can block training while still allowing search. The same split exists for Anthropic (ClaudeBot vs. Claude-SearchBot).
Does blocking Google-Extended remove my site from Google Search or AI Overviews?
No. Google states Google-Extended does not affect inclusion or ranking in Google Search. AI Overviews and AI Mode are part of Search and are controlled through Googlebot access and snippet directives such as nosnippet.
Why does a crawler show Allowed when I never mentioned it in robots.txt?
When no group names a crawler, it follows the User-agent: * group, and if nothing there matches the path, access is allowed by default. The row says which group applied so you can see the fallback.
My robots.txt allows a bot, but the server test says it is blocked. Why?
Something other than robots.txt, usually a CDN or firewall bot rule, is rejecting that user agent. Check your bot-management or WAF settings. Because our test request does not come from the crawler’s own IP addresses, confirm in your server logs before changing anything.
Is the robots.txt I paste uploaded anywhere?
No. Paste mode parses and evaluates the text in your browser. Only Live website mode contacts our server, which fetches the public robots.txt and page; results are returned to you and not stored.
Do ChatGPT-User and Perplexity-User follow robots.txt?
Not necessarily. OpenAI says robots.txt rules may not apply to ChatGPT-User because a person initiated the request, and Perplexity says Perplexity-User generally ignores robots.txt. Use authentication or server-side rules for content that must not be fetched.