Summary: Before deciding whether to block AI crawlers, separate them by purpose. OpenAI's OAI-SearchBot powers ChatGPT search, while GPTBot collects training data. Google-Extended controls whether content is used to train Gemini models and ground answers, but it does not affect Google Search inclusion or ranking. Sites that want visibility in AI search usually allow search crawlers and then decide separately on training crawlers.
First, decide which kind of crawling you want to refuse
Many sites block "AI bots" wholesale and accidentally switch off AI search visibility too. According to OpenAI's crawler overview, there are at least three kinds of access:
| User agent | Purpose | Effect of blocking | |---|---|---| | OAI-SearchBot | Surfaces websites in ChatGPT search results | Pages may not appear as links in ChatGPT search answers | | GPTBot | Crawls content for training generative models | Content is not used for training; ChatGPT search display is unaffected | | ChatGPT-User | Triggered when a user asks ChatGPT to visit a page | User-requested visits to your page may fail |
Perplexity likewise separates PerplexityBot (search results) from Perplexity-User (user-triggered visits); see the Perplexity crawler docs. Blocking AI crawlers is not one switch—it is a decision per user agent.
❌ Common mistake: turning on a CDN or site-builder "block AI" preset that also blocks search crawlers, then expecting ChatGPT search citations. ✅ Better approach: list the uses you want to refuse and write rules only for those user agents.
For export sites and SaaS: how to decide
Look at three things. First, is the content itself what you sell, such as paid reports or proprietary databases? If so, refusing training crawlers makes sense. Second, do your buyers use ChatGPT or Perplexity to research suppliers or software? If they do, blocking search crawlers removes you from those answers. Third, can you identify these user agents in server logs to confirm your rules work?
Marketing blogs, product pages and help docs exist to be read by prospects, so there is rarely a reason to block search crawlers there. Admin, order and quote pages should be protected by authentication; Google's robots.txt guide states robots.txt is not a mechanism for keeping pages private.
How to block GPTBot with robots.txt
To refuse training while keeping ChatGPT search visibility:
To protect specific directories only:
OpenAI notes that search-side changes take roughly 24 hours to take effect. Avoid broad wildcard patterns that match "bot", which can catch Googlebot or Bingbot.
What does Google-Extended do?
Per Google's common crawlers documentation, Google-Extended is a standalone product token without a separate crawling user agent. It controls whether content crawled by Googlebot may be used to train future Gemini models and to ground answers in products such as the Gemini app. It does not affect inclusion or ranking in Google Search.
AI Overviews and AI Mode are Search features governed by Googlebot and normal Search controls, not Google-Extended; see Google's AI features guidance. To limit snippets in Search and AI Overviews, use page-level directives such as nosnippet or max-snippet.
After allowing crawlers, how to check AI search visibility
Allowing crawlers only makes pages readable; it does not guarantee citations. After launch:
- Confirm successful (200) hits from OAI-SearchBot and PerplexityBot in server or CDN logs;
- Re-test a fixed set of real buyer questions, saving the prompt, date, original answer and source links;
- Record brand mentions and page citations separately; mark untested items as "not observed", not zero;
- Review visits and enquiries from AI platforms separately instead of counting all direct traffic as AI.
23SEOGEO's AI bot accessibility check shows each user agent's robots.txt status, and the GEO Authority Score includes adoption as a diagnostic dimension—it measures readiness, not actual citation rate.
FAQ
Does blocking AI crawlers hurt SEO rankings?
Blocking GPTBot or Google-Extended does not affect Google Search rankings. Accidentally blocking Googlebot or Bingbot does affect normal indexing.
Will I still appear in ChatGPT search after blocking GPTBot?
Yes. According to OpenAI, search display depends on OAI-SearchBot; blocking GPTBot alone does not remove you from search results.
How do I check whether ChatGPT cites my site?
Re-test fixed questions regularly and keep the original answers and source links, and look at visits from sources such as chatgpt.com in analytics. One re-test without a mention does not disprove earlier records; answers vary by time, region and account.
Official sources and author
- OpenAI: Overview of OpenAI Crawlers
- Google: Common crawlers and Google-Extended
- Google: AI features and your website
- Google: Introduction to robots.txt
- Perplexity: Perplexity Crawlers
Sources checked on 2026-11-06; this is our verification date, not the publication date of the documents. robots.txt snippets are configuration examples, not client case studies; images are editorial illustrations.
Author: Miss.Zuo, SEO article writer; published by the 23SEOGEO team. See the author profile at the top of the article.
