Meta robots directives reference
The robots meta tag and X-Robots-Tag header share a vocabulary of indexing directives. This page is a reference for the common ones — noindex, nofollow, noarchive, nosnippet, and the max-snippet family — explaining what each does and how to combine them.
Core directives
The most-used directives, valid in both the meta robots tag and X-Robots-Tag:
noindex — keep the page out of the search index. nofollow — do not follow links on the page for indexing. noarchive — do not show a cached link for the page. nosnippet — do not show a text snippet or video preview in results.
The defaults are the opposite (index, follow), so you only declare what you want to change.
- noindex — exclude from the index
- nofollow — do not follow the page's links
- noarchive — no cached copy link
- nosnippet — no result snippet or preview
Snippet limits and combining directives
Google documents finer snippet controls: max-snippet:[number] limits snippet length, max-image-preview:[setting] limits image preview size, and max-video-preview:[number] limits video preview length. For example:
<meta name="robots" content="max-snippet:120, max-image-preview:large">
You can combine compatible directives in one comma-separated value, for instance noindex, nofollow. Avoid pairing a removal directive (noindex) with conflicting consolidation signals, and remember that crawl access is the prerequisite for any of these to take effect.
- max-snippet, max-image-preview, max-video-preview tune previews
- Combine compatible directives in one comma-separated value
- The page must be crawlable for directives to be read
How it appears in analytics and logs
Each directive controls a different aspect of how a page appears in search. The crawler must fetch the page to read them, so a disallowed URL's directives are never applied.
Diagnostic use case
Look up the right directive to control indexing, link-following, snippet display, or caching for a page, and combine directives correctly.
What WebmasterID can help detect
WebmasterID shows whether crawlers still fetch a page, which is required for any of these directives to be read and acted on.
Common mistakes
- Declaring directives on a Disallowed URL so they are never read.
- Assuming nofollow on a page blocks crawling — it controls link-following for indexing.
- Combining noindex with conflicting consolidation signals.
Privacy and accuracy notes
These are public indexing signals, not access control. They shape how a page appears in search, not who can reach it.
Related pages
- The noindex meta tag
The noindex value of the meta robots tag tells search engines to keep a page out of their index. The catch trips people up constantly: for noindex to work, the crawler must be able to fetch the page — so you must not block the same URL in robots.txt.
- X-Robots-Tag header examples
X-Robots-Tag carries indexing directives in the HTTP response header instead of the HTML body, which makes it the way to apply noindex or nofollow to PDFs, images, and other non-HTML files. This page gives concrete header examples and notes how server config applies them in bulk.
- Canonical vs noindex: which to use
rel=canonical and noindex are often confused. Canonical tells search engines which of several similar URLs to treat as the primary, consolidating signals onto it. noindex removes a page from the index entirely. This page explains when each is right and why combining them on one URL sends conflicting signals.
- Website observability
Confirm a page is crawlable so its directives are read.
Sources and verification notes
- Google — Robots meta tag, data-nosnippet, and X-Robots-TagDocuments noindex, nofollow, noarchive, nosnippet, and max-snippet directives.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.