Crawl Budget
Crawl Budget is the upper bound on URLs Googlebot will fetch from your site in a given period, set by Crawl Rate Limit (server health) and Crawl Demand (page value). Small sites needn't worry; sites with 100k+ URLs must manage it actively.
Crawl Budget is the upper bound on URLs Googlebot will fetch from your site in a given period, set by Crawl Rate Limit (server health) and Crawl Demand (page value). Small sites needn't worry; sites with 100k+ URLs must manage it actively.
What is Crawl Budget?
Crawl Budget is the upper bound on URLs Googlebot will fetch from your site in a given period, set by Crawl Rate Limit (server health) and Crawl Demand (page value). Small sites needn't worry; sites with 100k+ URLs must manage it actively.
Details and examples
Save budget: block faceted-search parameters in robots.txt, noindex thin pages, fix 4xx/5xx errors, consolidate near-duplicate low-value pages.
Related terms
robots.txt, Indexation.
Crawl Budget · term at a glance
| Field | Value |
|---|---|
| Term | Crawl Budget |
| Category | Technical |
| Related | robots.txt, Indexation |
Frequently asked questions
What is Crawl Budget?
Crawl Budget is the upper bound on URLs Googlebot will fetch from your site in a given period, set by Crawl Rate Limit (server health) and Crawl Demand (page value). Small sites needn't worry; sites with 100k+ URLs must manage it actively.
References
By the 23SEOGEO team · Last updated: