Definition
Crawl budget is the number of URLs a search engine is willing and able to fetch from a site in a given period. Google describes it as the product of two things: the capacity your server can handle without slowing down, and the demand its systems have for your pages based on how popular and how fresh they are. It is not a quota you can buy, and for most stores it is not the constraint they think it is.
A crawl profile for a catalogue of forty thousand products
- URLs a crawler can reach
- Roughly 900,000, once filter combinations, sort orders and search result pages are counted
- URLs the store wants crawled
- Around 45,000: products, categories, a small set of filtered pages and editorial content
- Where fetches were going
- The majority landing on parameterised category views that repeat the same products in a different order
- Server response under load
- Slower on the heaviest category pages, which reduces how fast the crawler is willing to go
- Effect on new products
- Days between publishing an item and the first fetch of its page
- After the crawl paths were narrowed
- Fetches concentrated on products and categories, with new items picked up sooner
An illustrative profile. The Search Console crawl stats report shows fetches by response code, by file type and by purpose, which is where a real version of this table comes from.
Why it matters
Crawl budget matters when a store is large or fast-changing, and matters very little otherwise. Google has said most sites do not need to think about it, and the sites that do are the ones with very many URLs or with content that changes daily and needs to be recrawled to stay accurate. For an ecommerce store the practical concern is price and availability: an item whose page has not been fetched since it went out of stock, or since its price changed, is being represented in search results by information the store no longer stands behind. The second concern is discovery speed, since a crawler spending its fetches on filtered duplicates finds new products more slowly.
Where it goes wrong
- Worrying about it on a small catalogue, where a few thousand clean URLs are crawled comfortably and the time would be better spent on the pages themselves
- Trying to fix it with a sitemap alone, which helps discovery but does nothing about the crawl paths that generate the unwanted URLs in the first place
- Blocking crawling of pages that also need their tags read, since a URL disallowed in robots.txt cannot have its noindex or its canonical seen
- Ignoring server response time, which directly influences how fast a crawler is willing to fetch, so a slow store limits its own crawl capacity
- Leaving long redirect chains and pages returning errors in place, which consume fetches without ever producing a page anyone can use
Questions about crawl budget
How can a store tell whether crawl budget is a real problem?
Look at the crawl stats report in Search Console and at the server logs. If the majority of requests are landing on parameterised or filtered URLs rather than on products and categories, or if new products take a long time to be fetched for the first time, there is something to fix. If crawling is spread sensibly and new pages appear within a day or two, the effort belongs elsewhere.
Does site speed affect crawl budget?
It affects the capacity half of it. A server that responds quickly and without errors lets a crawler increase its rate, while slow responses and server errors cause it to back off to avoid causing harm. This is separate from the ranking effect of page experience: here the concern is simply how many pages can be fetched before the crawler decides the site has had enough.
Do product pages that never rank still consume crawl budget?
Yes, and that is usually fine. Every product a store sells deserves to be crawled and indexed so it can be found by its own name and appear in shopping surfaces. The waste is not in slow-moving products, it is in URLs that are not products at all: sorted views, empty filter combinations, internal search results and session-specific addresses.