What is a crawl error?
A crawl error is a failed visit. Search engines discover pages by following links and sitemaps, and a crawler like Googlebot requests each one the way a browser would. When the request does not return a usable page, that is a crawl error: a 404 because the page is gone, a 500 because the server fell over, a timeout because it was too slow, a redirect loop that never resolves, a page blocked by robots.txt or a noindex tag the developer forgot to remove, a DNS failure that makes the whole site unreachable, or a soft 404 where the server says fine and the page says nothing.
Google Search Console groups these under page indexing, with reasons for each page it could not index, and it is the first place to look. Site errors (DNS, server, robots.txt) affect everything; URL errors affect individual pages and are usually a longer, less urgent list.
Why it matters
Why crawl errors matter
Because a page that cannot be crawled cannot rank. The most valuable page on a site, with the best content and the strongest links, is invisible if the crawler is turned away at the door. Site-level errors are emergencies: a misconfigured robots.txt or a DNS failure can drop an entire site from search within days. Page-level errors are a slow leak: every broken link wastes crawl budget, every 404 that used to rank gives away the traffic and the link equity that pointed at it.
They also show up in the experience of real people. The links a crawler cannot follow are the links visitors cannot follow either, and a site full of dead ends looks abandoned. Crawl errors are usually the first visible symptom of a site that nobody is maintaining, and search engines treat that as the signal it is.
In practice
How crawl errors work in practice
- 01
Diagnosis starts in Search Console, where the page indexing report lists what failed and why, and the URL inspection tool shows exactly what the crawler saw for a single page. A crawling tool run against the site (Screaming Frog is the standard) finds the same problems from the outside, plus the broken internal links causing them. Server logs show the crawler's actual requests and responses, which is the ground truth when the reports disagree.
- 02
Fixes depend on the error. A page that moved gets a 301 redirect to its replacement; a page that is genuinely gone returns a proper 404 or 410 and gets removed from the sitemap and the internal links; a server error gets the server fixed or the slow query that caused it; a blocked page gets the robots rule or the noindex tag corrected; a redirect chain gets collapsed to a single hop. Then the fixed URLs are validated in Search Console so the crawler returns.
- 03
Prevention is maintenance. A crawl before and after every site change, especially a redesign or a migration, when most crawl disasters happen. A redirect map for every URL that changes. A sitemap that only lists pages that return 200. And someone whose job includes opening Search Console every week, because the report only helps if it is read.
- 04
The list is triaged, not worked top to bottom. Each failing URL is weighed by what it was worth, its traffic, the links pointing at it, the revenue it carried, so the home page returning an error is fixed within the hour and a stray old tag page waits. Fixes that need a developer go over as a ticket with the URL, the status code the crawler received, where the link lives and what the response should be.
Watch out
What people get wrong about crawl errors
Panicking over every 404. Some pages are supposed to be gone, and a 404 for a deleted product is correct; the problem is only a 404 for a page that should exist or that still has links pointing at it. The other mistake is the opposite: ignoring the report until the traffic drops. Site migrations without redirect maps are the most expensive version of this, and they happen every week to brands that thought the redesign was a design project.
A third error is confusing a crawl problem with a ranking problem. A page that is crawled, indexed and sitting on page four does not have a crawl error; it has a content or authority problem, and no amount of technical fixing will move it. Read the report for what it says, fix what it lists, and then look elsewhere for the rest.
