What Is Crawl Budget? Optimisation for Large Sites
Crawl budget is one of the most discussed SEO topics and one that few sites actually need to worry about. On a company site with a few hundred pages, budget is almost never the reason new pages are slow to be indexed.
On a store with hundreds of thousands of products, filter combinations and parameter URLs, the picture changes. Based on Google's crawl budget documentation, this article covers who it matters for, where waste comes from and how to read the Crawl Stats report.
Short answer: Crawl budget is the set of URLs Google can and wants to crawl on a site, shaped by server capacity and crawl demand. Google says it mainly matters for very large or very frequently changing sites. Optimisation means reducing unnecessary URLs, consolidating duplicates and keeping the server fast.
What is crawl budget and how does Google set it?
Google's documentation defines crawl budget through two components. The crawl capacity limit is how many connections Googlebot can open, and for how long, without overloading your server. It rises when the site responds quickly and healthily, and falls when Google sees slowdowns, 5xx errors or 429 responses.
Crawl demand is how much Google wants to crawl the site. The documentation names perceived URL inventory, popularity and staleness as the drivers. Even with enough capacity, low demand means less crawling.
Of the two, inventory is what you control most: the more unnecessary URLs you expose, the less time is left for the pages that matter.
Who does crawl budget matter for?
Google describes the audience for its guide roughly as follows, and stresses that these numbers are a rough estimate, not exact thresholds:
| Site profile | Does crawl budget matter? | What to check first |
|---|---|---|
| 1M+ unique pages, content changing weekly | Yes | Inventory, filter URLs, server speed |
| 10,000+ pages, content changing daily | Yes | Sitemaps, lastmod, freshness |
| Large share of URLs in "Discovered – currently not indexed" | Probably | Server response, duplicate URLs |
| A few hundred or a few thousand pages | Usually not | Content quality and internal links |
If a page on a small site is not indexed, the issue is most likely quality or uniqueness; the diagnosis steps in fixing crawled but not indexed pages are more useful there.
What wastes crawl budget the most?
In audits of large sites, these are the items that eat most of Googlebot's time:
The extreme case is launching hundreds of templated pages at once, which is why inventory control comes first in programmatic SEO projects. Links generated by JavaScript and rendering cost also affect crawling; see JavaScript SEO for details.
Crawl budget optimisation, step by step
Combining Google's recommendations with what works in practice, the process runs in this order:
- Map the inventoryUse server logs and a crawler to see which URL patterns Googlebot visits.
- Consolidate duplicatesGather URLs with the same content under one canonical address.
- Block the unimportantKeep sort variants and infinite-scroll duplicates out of crawling with robots.txt.
- Fix status codesReturn 404/410 for permanently removed pages, fix soft 404s, shorten chains.
- Speed up the serverLower response time and add caching and 304 (Not Modified) support.
A subtle but important point in the documentation: do not use noindex to save budget, because Google still requests the page and simply does not index it. For URLs you truly want left uncrawled, robots.txt is the right tool; see the robots.txt guide for syntax and pitfalls. When cleaning up chains, the redirect generator speeds up writing rules in bulk.
How to read the Crawl Stats report
The Crawl stats report under Settings in Search Console is only available for root-level properties (a Domain property or a root URL-prefix property). Google's help page says sites with fewer than a thousand pages should not need it.
- Total crawl requests and download size: sudden drops may signal access issues, sudden spikes a new source of URLs.
- Average response time: a rising curve can lead to a lower capacity limit.
- Host status: robots.txt fetch, DNS resolution and server connectivity problems show up here.
- Breakdown by response, file type and purpose: the discovery versus refresh split shows how quickly new URLs are found.
- Breakdown by Googlebot type: watch the smartphone crawler's share and page resource fetches.
Illustrative example, not data from a real site. A split like this comes from grouping the report's crawl examples or server logs.
If Googlebot spends almost half its time on filter URLs, filter architecture is the first job. The ecommerce SEO guide explains how to build faceted navigation, and the broader crawling and indexing framework is in the technical SEO guide.
When do you need expert help?
If URL Inspection shows a "Hostload exceeded" warning, Google's documentation recommends adding server resources. Log analysis, filter architecture and server-side settings usually need developers and SEO working together.
For stores with hundreds of thousands of URLs, I run this work as a template-level plan within ecommerce SEO consulting. To find out together whether budget is really a problem on your site, see SEO consulting or simply book a call.
Frequently Asked Questions
Does a sitemap increase crawl budget?
A sitemap does not enlarge the budget, but it helps Google find important URLs and notice changes. Google recommends using lastmod for updated content. Only canonical URLs you want indexed should be listed.
Does publishing more often increase crawling?
Staleness is one of the factors behind crawl demand. But merely changing dates or making meaningless edits has no lasting effect; Google cares about real changes and the page's value.
What should I do if Googlebot is overloading my server?
Google slows down when it sees 5xx and 429 responses. In an emergency, returning 500, 503 or 429 helps; Google advises limiting this to a few hours or 1-2 days, because errors lasting several days can get URLs removed from the index. The lasting fix is better server capacity and caching.
Do I need to look at server logs?
On large sites logs give the clearest picture, showing which URLs Googlebot visits and how often. To confirm requests really come from Google, use Google's published IP ranges or a reverse DNS check.
Does crawl budget affect rankings directly?
Being crawled is not a ranking factor, but a page that is not crawled cannot be indexed or ranked. That makes budget an indirect but important issue on large sites.
8+ years of hands-on SEO and GEO experience across 150+ projects in e-commerce, SaaS, healthcare and local business sectors. Writes based on real client data, not theory.