robots.txt
robots.txt is a crawler-instruction file at the site root using User-agent / Allow / Disallow to control who can fetch which paths. In the GEO era you must explicitly Allow GPTBot, PerplexityBot, and Google-Extended — otherwise AI engines will not cite you.
Category: Technical
What is robots.txt?
robots.txt is a crawler-instruction file at the site root using User-agent / Allow / Disallow to control who can fetch which paths. In the GEO era you must explicitly Allow GPTBot, PerplexityBot, and Google-Extended — otherwise AI engines will not cite you.
How do you apply it in practice?
Example snippet:
```
User-agent: GPTBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
```
Why does it matter for AI search citations?
Generative search engines tend to extract sentences that are clearly defined, consistent and quotable on their own. Writing concepts like "robots.txt" as one consistent definition across your site, backed by structured data and verifiable sources, lowers the chance of being misread. This improves readiness only; it does not guarantee citations or ranking gains — measure actual citations to know.
Which related terms should you know?
llms.txt
llms.txt is a plain-text manifest at a site's root that lists key pages, APIs, and docs in Markdown for AI crawlers to index efficiently. Proposed by Anthropic in 2024, it has become the de-facto 'robots.txt for the LLM era'.
GEO (Generative Engine Optimization)
GEO (Generative Engine Optimization) is the practice of optimizing content for generative AI search engines like ChatGPT, Perplexity, and Google AI Overviews so a brand gets cited inside AI answers. Core tactics include Answer Capsules, entity consistency, structured data, and AI-readable llms.txt files.
Crawl Budget
Crawl budget is the set of URLs Googlebot can and wants to crawl, influenced by crawl capacity and crawl demand rather than a fixed quota. It mainly matters for large or frequently updated sites. Check discovery and server health first: more crawling does not guarantee indexing or rankings.
robots.txt at a glance
| Term | robots.txt |
|---|---|
| Category | Technical |
| Last reviewed | 2026-10-06 |
References
- Google Search Central: SEO documentation
- Schema.org structured data vocabulary
- W3C JSON-LD 1.1 Recommendation
- GEO: Generative Engine Optimization (arXiv paper)
These documents support the definitions and methods, not ranking or citation promises; reviewed as of 2026-10-06.
By the 23SEOGEO team · Last updated: