X-Robots-Tag header examples
X-Robots-Tag carries indexing directives in the HTTP response header instead of the HTML body, which makes it the way to apply noindex or nofollow to PDFs, images, and other non-HTML files. This page gives concrete header examples and notes how server config applies them in bulk.
Common header examples
X-Robots-Tag uses the same directive vocabulary as the meta robots tag, delivered in the response header. Examples:
X-Robots-Tag: noindex X-Robots-Tag: noindex, nofollow X-Robots-Tag: noarchive
You can also scope a directive to a specific crawler by prefixing its product token:
X-Robots-Tag: googlebot: noindex
Google documents this header form and its supported directives.
- noindex — keep the resource out of the index
- noindex, nofollow — also do not follow its links
- googlebot: noindex — scope a directive to one crawler
Applying it to PDFs and in bulk
The header's main advantage is reaching files with no HTML head. To keep PDFs out of search, set the header for that content type in your server config — for example matching .pdf requests and adding X-Robots-Tag: noindex — so every PDF is covered without editing each file.
Because the directive lives in the response, you can apply it server-wide or per-path. The catch is the same as for meta robots: the crawler must fetch the resource to read the header, so do not also Disallow the URL in robots.txt.
- Set the header by content type to cover all PDFs at once
- Apply server-wide or per-path via configuration
- Do not also Disallow the URL, or the header is never read
How it appears in analytics and logs
An X-Robots-Tag in a response controls indexing of that resource. As with meta robots, the crawler must be able to fetch the resource to read the header — a disallowed URL's header is never seen.
Diagnostic use case
Apply noindex, nofollow, or noarchive to non-HTML resources and to many URLs at once through server or edge configuration.
What WebmasterID can help detect
WebmasterID shows whether crawlers still fetch a resource, helping you confirm the crawler can reach it to read the X-Robots-Tag you set.
Common mistakes
- Disallowing a PDF in robots.txt so its X-Robots-Tag noindex is never read.
- Expecting X-Robots-Tag to block crawling — it controls indexing only.
- Editing files one by one instead of setting the header by content type.
Privacy and accuracy notes
X-Robots-Tag is an indexing signal, not access control. The resource still needs authentication to be truly private.
Related pages
- robots.txt vs the X-Robots-Tag header
X-Robots-Tag carries the same indexing directives as the meta robots tag, but in the HTTP response header instead of the HTML body. That makes it the way to apply noindex or nofollow to non-HTML resources like PDFs and images, where a meta tag has nowhere to live.
- Meta robots directives reference
The robots meta tag and X-Robots-Tag header share a vocabulary of indexing directives. This page is a reference for the common ones — noindex, nofollow, noarchive, nosnippet, and the max-snippet family — explaining what each does and how to combine them.
- The noindex meta tag
The noindex value of the meta robots tag tells search engines to keep a page out of their index. The catch trips people up constantly: for noindex to work, the crawler must be able to fetch the page — so you must not block the same URL in robots.txt.
- Website observability
Confirm crawlers can reach the resources you annotate.
Sources and verification notes
- Google — Robots meta tag and X-Robots-TagDocuments X-Robots-Tag header syntax, directives, and per-crawler scoping.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.