llms.txt
llms.txt is a plain-text manifest at a site's root that lists key pages, APIs, and docs in Markdown for AI crawlers to index efficiently. Proposed by Anthropic in 2024, it has become the de-facto 'robots.txt for the LLM era'.
Category: Technical
What is llms.txt?
llms.txt is a plain-text manifest at a site's root that lists key pages, APIs, and docs in Markdown for AI crawlers to index efficiently. Proposed by Anthropic in 2024, it has become the de-facto 'robots.txt for the LLM era'.
How do you apply it in practice?
Format:
```
# Site Name
> One-line description.
## Core Pages
- [Pricing](https://...): one-line summary
```
llms-full.txt is the extended version — concatenate each page's core paragraph so an AI can understand the site without crawling.
Why does it matter for AI search citations?
Generative search engines tend to extract sentences that are clearly defined, consistent and quotable on their own. Writing concepts like "llms.txt" as one consistent definition across your site, backed by structured data and verifiable sources, lowers the chance of being misread. This improves readiness only; it does not guarantee citations or ranking gains — measure actual citations to know.
Which related terms should you know?
GEO (Generative Engine Optimization)
GEO (Generative Engine Optimization) is the practice of optimizing content for generative AI search engines like ChatGPT, Perplexity, and Google AI Overviews so a brand gets cited inside AI answers. Core tactics include Answer Capsules, entity consistency, structured data, and AI-readable llms.txt files.
robots.txt
robots.txt is a crawler-instruction file at the site root using User-agent / Allow / Disallow to control who can fetch which paths. In the GEO era you must explicitly Allow GPTBot, PerplexityBot, and Google-Extended — otherwise AI engines will not cite you.
AI Citation
AI Citation refers to source links or quoted sentences that ChatGPT, Perplexity, and Google AI Overviews surface in their answers. Citation Frequency has become the core brand-visibility metric in the GEO era, replacing rank-and-click as the headline KPI.
llms.txt at a glance
| Term | llms.txt |
|---|---|
| Category | Technical |
| Last reviewed | 2026-10-06 |
References
- Google Search Central: SEO documentation
- Schema.org structured data vocabulary
- W3C JSON-LD 1.1 Recommendation
- GEO: Generative Engine Optimization (arXiv paper)
These documents support the definitions and methods, not ranking or citation promises; reviewed as of 2026-10-06.
By the 23SEOGEO team · Last updated: