AI Bot Accessibility — Does robots.txt Allow ChatGPT, Claude & Perplexity? | SEO-GEO
Plenty of sites are ignored by AI engines not because the content is weak, but because robots.txt quietly blocks newer agents like GPTBot or Bytespider. This checker applies RFC 9309 rules line by line — no guessing.
The tool reads only /robots.txt at the target origin, groups rules by User-agent, and applies RFC 9309 longest-path precedence to decide whether the home path is allowed. Agents absent from robots.txt are reported as Unconfigured — the protocol default is allow, but engines differ in practice, so an explicit rule is safer.
What can you check or produce?
- Covers 10 major AI crawlers: OAI-SearchBot / GPTBot / anthropic-ai / ClaudeBot / PerplexityBot / Google-Extended / Bytespider and more
- Three states: Allowed / Blocked / Unconfigured (no rule for that agent)
- Parses Allow and Disallow with RFC 9309 longest-match precedence
- Generates a copy-paste robots.txt fix block for every blocked agent
- Chinese-market crawlers such as Bytespider are reported separately
- Login-free, fetch-only, and consumes no AI credits
How should you interpret results and choose the next action?
The tool reads only /robots.txt at the target origin, groups rules by User-agent, and applies RFC 9309 longest-path precedence to decide whether the home path is allowed. Agents absent from robots.txt are reported as Unconfigured — the protocol default is allow, but engines differ in practice, so an explicit rule is safer. Allowed means allowed at the robots.txt layer only. CDN rules, WAFs, UA blocklists and Cloudflare Bot Fight can still block the fetch higher up. To verify real reachability, follow up with the AI Crawler Fetch Tester, which issues requests using the real user agents. Allowing AI crawlers is necessary for citation, not sufficient. Whether to allow training-oriented agents like GPTBot or Google-Extended is a business decision; the tool reports the current state and the fix, and does not decide for you.
Frequently asked questions
Does Unconfigured matter?
The protocol default is allow, but engines handle undeclared agents inconsistently, so write explicit rules for the AI crawlers you want in.
Everything is allowed but I still get no citations. Why?
robots.txt is only the doorway. Rendering, structured data and actual citation-worthiness all matter — run the GEO Authority Score next.
Should I allow GPTBot?
That is a business decision: allowing helps training coverage, blocking protects your content asset. The tool shows the state and the fix without taking a side.
By the 23SEOGEO team · Last updated: