What is Googlebot?
Googlebot is the name of the software that visits web pages on Google's behalf. It follows links and sitemaps, requests pages the way a browser would, renders them to see what a visitor would see, and hands the result to Google's indexing systems. There are two main versions, a smartphone crawler and a desktop crawler, and since mobile-first indexing the smartphone one does most of the work. There are also specialist crawlers for images, video, news and ads, all under the same family name.
It identifies itself in the user agent string, it obeys the instructions in a site's robots.txt file, and it is polite by design: it paces its requests so that it does not overload a server, and it backs off when a site responds slowly or with errors.
Why it matters
Why Googlebot matters
Because nothing ranks that Googlebot has not seen. A page it cannot reach, cannot render or has been told to ignore does not exist as far as Google search is concerned. That makes the crawler the first audience for every page on a site, and a demanding one: it has a limited budget of attention for each site, it does not click, scroll or log in, and it has no patience for pages that take seconds to assemble.
It also matters as a question of control. The same instructions that keep Googlebot out of a staging site or a search results page can, with one wrong line, keep it out of a launch. Knowing how the crawler reads a site is how you avoid that.
In practice
How Googlebot works in practice
- 01
Googlebot keeps a list of known URLs and schedules visits based on how important a page seems and how often it changes. On each visit it fetches the HTML, then queues the page for rendering, where it runs the JavaScript and builds the page much as Chrome would. The rendered version is what gets indexed, which is why content that only appears after a click, a scroll or a delayed script may never be seen. Links found along the way are added to the list, and a sitemap gives it a shortcut to pages links alone might miss.
- 02
On the site side, the levers are robots.txt for what it may crawl, meta robots tags and canonicals for what it should index, the sitemap for what you want it to prioritize, and server speed for how much it can get through. Server logs show exactly which pages it is visiting and how often, which is the most honest crawl report there is.
At RIOT
Googlebot at RIOT
AltSounds, our music video channel, is a large catalog site, and the questions on this page are ones we deal with there in a practical way: which pages Googlebot should spend its time on, which filter and sort URLs it should skip, and how to tell the difference between a real crawl and something impersonating one.
Watch out
What people get wrong about Googlebot
Assuming it sees what a visitor sees. It does not hover, it does not accept cookies, it does not wait for a lazy-loaded gallery, and it does not follow a link that is really a button with a script attached. The other mistake is treating robots.txt as a privacy tool: a blocked URL can still be indexed from links elsewhere, just without its content. The third is trusting the user agent string alone. Anything can call itself Googlebot, and verifying a crawler means checking that the request really came from Google's published address ranges.
