Crawl budget

Crawl budget is how many of your URLs a search engine will fetch in a given window. It is a function of crawl capacity (how much your server can handle without slowing) and crawl demand (how much the engine wants your content). For most small sites it is a non-issue; for large sites it decides how quickly new and updated pages get discovered and indexed.

Why it matters

When a site has millions of URLs, a crawler will not fetch them all often. If it burns its budget on low-value pages — infinite faceted-navigation combinations, session-ID URLs, redirect chains, soft 404s — your important pages get crawled and refreshed less. Managing crawl budget means removing waste (block or consolidate junk URLs), keeping the site fast, and pointing crawlers at what matters through clean internal links and sitemaps. The goal is not “more crawling” but crawling spent on the right pages.

Where it applies

Large or fast-changing sites feel it most:

• Enterprise ecommerce — millions of SKU, facet, and parameter URLs competing for a finite budget.
• Marketplaces and classifieds — high URL churn where fresh listings must be found fast.
• News and publishing — time-sensitive content that loses value if crawled late.

What matters from each seat

• In-house — audit what crawlers actually spend budget on, then prune the low-value URL space.
• Engineering — server speed and clean status codes directly raise crawl capacity; fix redirect chains and 5xx errors.
• Agency and consulting — a log-file crawl audit is the credible way to prove where budget leaks on a large site.