What is crawl budget?
Googlebot does not crawl everything, every day. For each site it balances two things: a crawl capacity limit, which is how fast it can fetch pages without slowing the server for real visitors, and crawl demand, which is how much it wants to, based on how popular the pages are and how often they change. The result is a budget, a rough number of URLs it will visit in a day, and on a site with more URLs than budget, some pages are visited rarely and some never. Google has said sites under a few thousand pages need not worry; sites with hundreds of thousands of URLs do.
The budget is spent on whatever the crawler finds, which includes the URLs you did not mean to exist: filter combinations, session parameters, calendar pages that go on forever.
Why it matters
Why crawl budget matters
Because a page that is not crawled is not indexed, and a page that is crawled once a quarter is updated in the index once a quarter. On a large e-commerce or media site, budget wasted on duplicates and junk means new products and fresh articles wait days to appear in search, and the long tail of the site drifts out of date.
On a smaller brand site it matters less as a quantity and more as a diagnosis: when Search Console shows thousands of crawled-not-indexed URLs on a site that has two hundred real pages, something is generating addresses nobody designed.
In practice
How crawl budget works in practice
- 01
Find out where it goes. Server log analysis shows which URLs Googlebot actually fetches, how often and with what status; Search Console's crawl stats report shows the totals, the response times and the share by file type. Together they reveal the waste: parameter URLs, redirect chains, 404s being retried, paginated archives, staging copies, infinite calendars.
- 02
Then cut the waste and raise the ceiling. Block pointless URL patterns in robots.txt, canonicalize variants, return 410 for dead pages instead of letting them be retried, fix redirect chains, and keep the XML sitemap limited to the URLs that should be indexed. Raise capacity by making the server faster, since a quicker response lets Googlebot fetch more in the same time. Then watch the crawl stats to see the budget move toward the pages that matter.
At RIOT
Crawl budget at RIOT
riot.nyc is a small site by crawl standards. Muvvoo, our music video database, and AltSounds, our music video channel, are not: a database with a page for every video and every artist generates far more URLs than any crawler will visit in a day, and every filter and sort order threatens to multiply them. Crawl budget is a daily discipline on those sites, and the lesson carries back to the brand work: decide which URLs deserve to exist, and make the rest impossible to find.
Watch out
What people get wrong about crawl budget
Worrying about it on a fifty-page site. Google will crawl a small site completely without help, and the hours spent "optimizing crawl budget" there would be better spent on the content. The opposite error is on large sites: assuming the sitemap is what Googlebot crawls, when in fact it crawls what it finds, and most of what it finds on a big site is links the sitemap never mentioned. Blocking a pattern in robots.txt stops the crawling but does not remove already-indexed URLs; for that you need noindex or 410 and the patience to let it be recrawled.
