What is the X-Robots-Tag?
The X-Robots-Tag is a line in the HTTP headers a server sends with a response, carrying the same directives as the robots meta tag: noindex, nofollow, noarchive, nosnippet, noimageindex, unavailable_after and the rest. A robots meta tag lives in the HTML of a page. The X-Robots-Tag lives outside the content, in the header, so it can be attached to anything the server delivers, including PDFs, images, videos, spreadsheets and plain text files that have no head section to put a tag in.
It can also be applied to HTML pages, and is often used that way when the rule needs to cover a whole folder or file type at once. A single server configuration line can noindex every PDF on a site, which would otherwise take a change to each file or be impossible.
Why it matters
Why the X-Robots-Tag matters
Sites leak through their files. Internal documents, draft PDFs, old brochures, high-resolution images, exported spreadsheets: all of them are indexable, and many end up in search results because nobody thought of them as pages. The X-Robots-Tag is the only way to keep them out without deleting them or hiding them behind a login. It is also the correct tool for the job that robots.txt is often misused for: robots.txt stops crawling, which does not stop indexing, while a noindex header stops indexing directly.
It matters at scale. A site with thousands of media files or a staging environment can set a single rule at the server and be sure nothing slips through, instead of maintaining tags file by file.
In practice
How the X-Robots-Tag works in practice
- 01
The header is set in the server configuration or the application. On Apache it is a line in .htaccess matching a file pattern; on Nginx a directive in the server block; on a CDN or edge platform a rule on the response. A typical rule sends "X-Robots-Tag: noindex" for every URL ending in .pdf, or "noindex, nofollow" for an entire staging subdomain. Content management systems and SEO plugins can set it for specific URLs or types.
- 02
Testing is essential, because a header is invisible in the page. The response headers are checked with browser developer tools, curl or the URL Inspection Tool, which reports the directives it found. The rule has to allow crawling: a file blocked in robots.txt will never have its header read, so the noindex has no effect. And it is audited, because a header set broadly during a migration, such as noindex on everything, has taken sites out of the index when it was forgotten at launch.
Watch out
What people get wrong about the X-Robots-Tag
Combining it with a robots.txt block. If the crawler cannot fetch the file, it cannot see the header, and the file may stay indexed from links alone. The other mistake is the inverse: forgetting that it exists. Sites with a perfect set of meta tags on their pages are surprised to find their PDFs ranking, because nobody applied the equivalent rule to files. And the launch-day error is common enough to be a cliche: a staging rule of noindex that ships to production and quietly removes the site from search.
