What is a search engine index?
The index is the search engine's library. When a crawler fetches a page, the search engine renders it, works out what it is about, extracts its text, images, links and structured data, and files all of that under the words and concepts the page relates to. That filing system is the index, and a search query is answered by looking things up in it rather than by searching the live web. Google's index holds hundreds of billions of pages and is the largest such library ever built.
A page is indexed when it has been through that process and stored. A page that has been crawled but judged a duplicate, blocked by a noindex tag, or found to be thin or broken is not. The difference is the whole game: an unindexed page cannot rank for anything, however good it is.
Why it matters
Why the index matters
Because it is the gate before every other SEO effort. Keyword research, content, links and speed all presume the page is in the index to begin with, and on real sites a surprising share of pages are not. Search Console's Pages report routinely shows thousands of URLs that were crawled and left out, and some of them are pages the business cares about: a new product line, a campaign landing page, a case study that took weeks to write.
It also matters in the other direction. Pages a brand would rather not have in the index (staging copies, internal search results, thin tag archives, expired promotions) sit there competing with the pages that matter, and part of looking after a site is deciding what should be in the library and what should not.
In practice
How the index works in practice
- 01
Googlebot discovers a URL through links or a sitemap, crawls it, and queues it for rendering and indexing. At that stage Google decides whether the page is worth storing: whether it is the canonical version or a duplicate, whether the content is substantial, whether a robots directive forbids it. Indexed pages are then re-crawled on a schedule that depends on how often they change and how important they seem. The index is not static; pages fall out of it when they go stale or break, and go back in when they are fixed.
- 02
The site owner's tools are a clean sitemap of the pages that matter, canonical tags that say which version of a page is the real one, noindex on pages that should stay out, and the URL Inspection tool for checking any single page's status and asking for a re-crawl. Searching Google with the site operator gives a rough count of what is in; Search Console gives the real one.
Watch out
What people get wrong about the index
Confusing crawled with indexed. Googlebot visiting a page does not mean the page was stored; the Pages report in Search Console separates the two, and the gap between them is where most indexing problems hide. The other mistake is thinking more indexed pages is better. An index full of near-duplicates and thin archives dilutes a site's signal, and pruning is often what lifts the pages that matter. The third is blocking a page in robots.txt to keep it out of the index, which prevents the crawl but not the indexing, so the page can still appear as a bare URL with no description.
