robots.txt
robots.txt is a crawler-instruction file at the site root using User-agent / Allow / Disallow to control who can fetch which paths. In the GEO era you must explicitly Allow GPTBot, PerplexityBot, and Google-Extended — otherwise AI engines will not cite you.
robots.txt is a crawler-instruction file at the site root using User-agent / Allow / Disallow to control who can fetch which paths. In the GEO era you must explicitly Allow GPTBot, PerplexityBot, and Google-Extended — otherwise AI engines will not cite you.
What is robots.txt?
robots.txt is a crawler-instruction file at the site root using User-agent / Allow / Disallow to control who can fetch which paths. In the GEO era you must explicitly Allow GPTBot, PerplexityBot, and Google-Extended — otherwise AI engines will not cite you.
Details and examples
Example snippet: ``` User-agent: GPTBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / ```
Related terms
llms.txt, GEO (Generative Engine Optimization), Crawl Budget.
robots.txt · term at a glance
| Field | Value |
|---|---|
| Term | robots.txt |
| Category | Technical |
| Related | llms.txt, GEO (Generative Engine Optimization), Crawl Budget |
Frequently asked questions
What is robots.txt?
robots.txt is a crawler-instruction file at the site root using User-agent / Allow / Disallow to control who can fetch which paths. In the GEO era you must explicitly Allow GPTBot, PerplexityBot, and Google-Extended — otherwise AI engines will not cite you.
References
By the 23SEOGEO team · Last updated: