Crawl Budget
Crawl budget is the set of URLs Googlebot can and wants to crawl, influenced by crawl capacity and crawl demand rather than a fixed quota. It mainly matters for large or frequently updated sites. Check discovery and server health first: more crawling does not guarantee indexing or rankings.
Category: Technical
What is Crawl Budget?
Crawl budget is the set of URLs Googlebot can and wants to crawl, influenced by crawl capacity and crawl demand rather than a fixed quota. It mainly matters for large or frequently updated sites. Check discovery and server health first: more crawling does not guarantee indexing or rankings.
How do you apply it in practice?
Start by separating discovery, crawling and indexing. In Search Console, 'Discovered – currently not indexed' means Google knows the URL but has not crawled it yet; that status alone does not prove a budget shortage. 'Crawled – currently not indexed' means crawling has already happened. Investigate content value, duplicates and canonical selection instead of assuming that more requests will solve the problem.
Capacity and demand are different constraints. Slow responses and server errors can reduce what Googlebot can crawl. Demand also depends on page importance, changes and the inventory of URLs Google knows about. Do not present the two as an additive fixed quota or use one URL-count threshold as a universal rule.
Collect evidence before changing access rules. Compare host status, response times and response-code patterns in Crawl Stats with verified Googlebot server logs. Record the date range and the last crawl of business-critical URLs. Repeated requests for parameter variants alongside neglected important pages can justify an investigation; a one-day decline alone is not evidence of a penalty.
Remove avoidable technical waste. Keep intended canonical URLs in your sitemap, link directly to final destinations and shorten unnecessary redirect chains. Fix broken URLs still referenced by navigation or sitemaps. A genuinely removed page may correctly return 404 or 410; redirect only when a relevant replacement exists. Do not change sitemap lastmod dates unless the page actually changed.
Review faceted navigation carefully. Sorting, price and color combinations can create a large near-duplicate URL space. Identify combinations with independent search demand before blocking them. Robots.txt limits crawling but is not a reliable removal-from-index mechanism. Noindex must remain crawlable to be read, and those pages still consume crawl requests. Canonical is a consolidation signal, not a promise that duplicate crawling stops immediately.
Recheck using a consistent measurement window. Track discovery, crawling and indexing of important pages alongside errors and unnecessary parameter requests. Improved allocation demonstrates a technical change, not guaranteed traffic growth. Google's guidance concerns Googlebot; it does not define another AI engine's behavior or predict citation probability.
Why does it matter for AI search citations?
Generative search engines tend to extract sentences that are clearly defined, consistent and quotable on their own. Writing concepts like "Crawl Budget" as one consistent definition across your site, backed by structured data and verifiable sources, lowers the chance of being misread. This improves readiness only; it does not guarantee citations or ranking gains — measure actual citations to know.
What are the common misconceptions?
- Crawl budget is not a fixed quota or a ranking promise.
- Noindex must be crawled to be read; it does not instantly save crawling.
- Blocking crawling and removing a URL from the index are different actions.
Which related terms should you know?
robots.txt
robots.txt is a crawler-instruction file at the site root using User-agent / Allow / Disallow to control who can fetch which paths. In the GEO era you must explicitly Allow GPTBot, PerplexityBot, and Google-Extended — otherwise AI engines will not cite you.
Indexation
Indexation is the process by which a search engine adds a page to its searchable index. Discovered ≠ Crawled ≠ Indexed ≠ Ranked. Google Search Console's Pages report is the diagnostic starting point; the IndexNow protocol (Bing/Yandex) lets you push URLs for faster indexing.
Crawl Budget at a glance
| Term | Crawl Budget |
|---|---|
| Category | Technical |
| Last reviewed | 2026-10-11 |
References
These documents support the definitions and methods, not ranking or citation promises; reviewed as of 2026-10-11.
By the 23SEOGEO team · Last updated: