What is crawl delay?
Crawl delay is a line in a site's robots.txt file that tells a crawler how long to pause between page requests. A search engine's bot does not read a site the way a person does; it fetches thousands of URLs in sequence, and on a small or busy server that can look like a traffic spike. The crawl-delay directive, written as a number of seconds, asks the bot to slow down: a value of 10 means one request every ten seconds. It was invented in the early days of search, when servers were weaker and crawlers were cruder, and it is still honored by some engines.
Not by Google, though. Googlebot ignores crawl-delay in robots.txt and manages its own pace based on how the server responds, with a manual rate limit available in Google Search Console for emergencies. Bing, Yandex and some smaller crawlers do respect it, which is why the directive still appears in robots.txt files that have been copied from one site to the next for twenty years.
Why it matters
Why crawl delay matters
Crawling costs a site something: every request uses server time, and a crawler that hits a weak server too hard can slow it for real visitors or take it down. For large sites with millions of pages, or small sites on cheap hosting, managing crawl rate is part of keeping the site up. Crawl delay is the oldest tool for that, and on the engines that respect it, the simplest.
It matters in the other direction too. Set a crawl delay too high and a site tells search engines to look at it rarely, which means new pages take longer to be found and changes take longer to be reflected. A site that publishes daily and asks Bing to visit once every thirty seconds has made itself hard to index. The directive is a throttle, and like any throttle it can be set too tight.
In practice
How crawl delay works in practice
- 01
The directive is added to robots.txt under a user-agent line, so it can apply to one crawler or all of them. A typical entry pairs a user-agent with a crawl-delay value of a few seconds. Engines that honor it will space their requests accordingly; Google will ignore it and should be managed through Search Console's crawl rate settings or, better, through a server that responds quickly enough that Google never needs slowing down. For aggressive crawlers that respect nothing, the practical tools are server-level rate limiting, a firewall or a content delivery network with bot management.
- 02
Most well-hosted sites do not need a crawl delay at all. The sensible process is to look at the server logs first, see which bots are actually causing load, and respond to those specifically, with a delay for engines that honor one and a block or a rate limit for those that do not. The robots.txt file should be reviewed when it is changed, because a stray directive copied from an old template can quietly throttle a search engine for years.
Watch out
What people get wrong about crawl delay
Assuming it affects Google. Site owners add a crawl delay to fix a load problem, see no change, and conclude the problem is elsewhere, when Google simply never read the line. If Googlebot is the load, the answer is a faster server or a Search Console rate limit, not robots.txt.
The other mistake is setting it and forgetting it. A delay added during a server emergency years ago is still slowing Bing's crawl today, and nobody remembers why it is there. Robots.txt is a file worth reading once a year.
A third is using crawl delay to fight bots that are not search engines. Scrapers, AI training crawlers and bad actors mostly ignore robots.txt altogether, so a delay aimed at them does nothing except slow the engines that were playing fair. Those need to be handled at the server or the firewall, where the request can actually be refused.
