What is a user-agent?
Every request to a web server carries a user-agent header: a line of text that says what kind of software is asking. A browser sends a long string that identifies the browser, its version and the operating system. Google's crawler sends a string containing the word Googlebot and a version number. Other search engines, social platforms fetching link previews, monitoring services, scrapers and AI crawlers all identify themselves with their own strings, and a server can read the header and decide how to respond.
The term also means the software itself. A browser is a user-agent; so is a crawler. In SEO the word is used mostly in the second sense, as in "which user-agents are hitting the site", and in the context of robots.txt, where rules are written per user-agent.
Why it matters
Why a user-agent matters
It is the basis of nearly every rule a site applies to crawlers. Robots.txt rules begin with a user-agent line that names who they apply to, so a site can allow Googlebot everywhere, keep an aggressive scraper out entirely and slow a third crawler down. Server logs are sorted by user-agent to see what the search engines are actually fetching and how often, which is the only direct evidence of crawl behavior a site has. Analytics filters out bots by user-agent so that reported traffic is people.
It also matters for rendering. Some sites serve different markup to crawlers, legitimately for performance or illegitimately as cloaking, and the user-agent is the switch. Knowing which user-agents a site treats differently, and why, is part of any technical audit.
In practice
How a user-agent works in practice
- 01
Log file analysis is the main use. The server's access logs are filtered to the user-agents of the major crawlers, and the results show which pages are crawled, which are ignored, how much of the budget goes to junk URLs, and whether a crawler is hitting errors the browser never sees. Because user-agents can be forged, serious analysis verifies that a request claiming to be Googlebot really came from Google's published address ranges.
- 02
The second use is control. Robots.txt rules are written per user-agent, with a default block for everyone and specific allowances for the crawlers the site wants. Firewall and hosting rules block or rate-limit user-agents that cause problems. Testing tools, including Google's own, let a page be fetched with a chosen user-agent so that the team can see exactly what each crawler receives.
Watch out
What people get wrong about user-agents
Trusting the string. Anyone can send any user-agent, and bad bots often claim to be Googlebot to slip past rules; verification by IP is the only reliable check. The other mistake is blocking by user-agent too freely, which can shut out a search engine's newer crawler, a social platform's preview fetcher or a partner's monitoring tool. And teams forget that analytics bot filtering depends on user-agent lists that need updating, so new crawlers quietly inflate traffic numbers until someone notices.
