What is robots.txt?
Robots.txt is the oldest instruction a website gives a search engine. It is a text file that lives at one address, the domain followed by /robots.txt, and it contains rules in a simple format: which crawler the rule is for, what it may not fetch, what it may, and where the sitemap is. The convention dates from 1994 and was formalized as a standard in 2022. Every well-behaved crawler checks the file before touching anything else on the site.
The file is advisory. It tells a crawler not to fetch a path; it cannot stop one that chooses to ignore it, and it does not hide the page from the index. If other pages link to a URL that robots.txt blocks, Google can still list the URL in results, with no description, because it knows the address exists even though it has not read the page. Keeping a page out of search is the job of a noindex tag, which the crawler has to be allowed to fetch in order to read.
Why it matters
Why robots.txt matters
It is the first thing a crawler reads, and one wrong line can take a site out of search. A single Disallow: / left over from the staging environment blocks everything, and sites have gone dark at launch for exactly that reason. The reverse mistake, blocking the CSS and JavaScript files that render the page, leaves the crawler seeing an unstyled, half-empty version and ranking accordingly.
Used well, it protects crawl budget. Large sites generate endless URLs from filters, internal search, calendars and session parameters, and a crawler left to wander them wastes its visit. A few lines in robots.txt keep it on the pages that matter. It has also become the place where sites state their position on AI crawlers, which now check it for permission to collect training data.
In practice
How robots.txt works in practice
- 01
The file is written by hand or generated by the content management system and placed at the root. Rules are grouped by user agent, with an asterisk for all crawlers and specific names for specific ones. Disallow lines block paths; Allow lines open exceptions inside them; a Sitemap line points to the XML sitemap. Google reads only the group that matches its crawler most specifically, so a rule for all crawlers does not apply to Googlebot if there is a Googlebot group.
- 02
Testing matters more than writing. Search Console has a robots.txt report that shows the file Google last fetched and any errors, and a URL inspection tool that says whether a given page is blocked. Before launch and after any change to the file, check the key pages: the homepage, the main sections, and the asset folders the design depends on. Keep the file short, comment it, and put it in version control, because the person who edits it next will not remember why the rules are there.
Watch out
What people get wrong about robots.txt
Using it to hide pages. A blocked URL can still appear in search results if anything links to it, and the block prevents Google from reading the noindex tag that would actually remove it. To keep a page out of the index, allow the crawl and add noindex.
The other mistake is forgetting it is public. Anyone can read a site's robots.txt, and a Disallow line for a secret section is an invitation to look there. It is a crawl instruction, not security.
