What is crawlability?
Before a page can rank it has to be found, and before it can be found a crawler has to be able to get to it. Crawlability is that property of a site: whether the crawler is allowed in (robots.txt, firewalls, login walls), whether it can follow the links from one page to the next (links in HTML rather than hidden behind scripts or forms), whether the server answers (status codes, speed, uptime) and whether what it fetches contains the content (rendered HTML rather than an empty shell waiting for JavaScript). Indexability is the next step, whether a crawled page is allowed into the index. Crawlability is the door; indexability is the room.
Most crawlability problems are accidents. Nobody decides to hide the product pages from Google; a developer adds a blanket disallow during the build and forgets to remove it.
Why it matters
Why crawlability matters
Because every other investment in search depends on it. The best content on the web does not rank if the crawler cannot reach it, and a site can lose whole sections, the case studies, the archive, the regional pages, without anyone noticing, because the people who run it navigate it with a mouse and the crawler navigates it with links. The symptom is pages that never appear in search, or disappear after a redesign, and the cause is usually a line of configuration.
It also compounds. A page the crawler cannot reach is also a page whose links are never followed, so the pages behind it are lost as well, and the ones with no links at all become orphan pages.
In practice
How crawlability works in practice
- 01
Audit it with a crawler of your own. Screaming Frog or Sitebulb follow links the way Googlebot does and report what they could not reach, what was blocked, what redirected and what returned errors. Compare that with Search Console's page indexing report, which lists URLs Google discovered but did not crawl or could not index, with reasons. Orphan pages show up as the gap between the sitemap and the crawl.
- 02
Then remove the obstacles. Open robots.txt to everything that should be indexed, link every page from at least one other page (and the important ones from many), keep the navigation in HTML, make sure JavaScript-rendered content also appears in the served HTML or is rendered server-side, fix the errors and submit a clean XML sitemap so the crawler has a map as well as the roads.
At RIOT
Crawlability at RIOT
Every entry in this glossary links to its related terms, so a crawler that lands on any one term can walk to the rest without a sitemap. It is the simplest crawlability device there is, a link, used deliberately, and it is the same principle we apply to a client's site: nothing worth indexing should be more than a few clicks from the homepage, and every page should have at least one road in.
Watch out
What people get wrong about crawlability
Confusing it with indexability. A page can be perfectly crawlable and carry a noindex tag, or be blocked in robots.txt and still appear in the index from links alone, as a bare URL with no description. Blocking a page you want removed is the wrong tool; let it be crawled so the noindex can be seen. The other mistake is trusting the sitemap: a sitemap tells the crawler a page exists, but a page with no links to it reads as unimportant, and the crawler treats it that way.
