The noindex meta tag
The noindex value of the meta robots tag tells search engines to keep a page out of their index. The catch trips people up constantly: for noindex to work, the crawler must be able to fetch the page — so you must not block the same URL in robots.txt.
How to use noindex
Add the meta robots tag with a noindex value in the page's <head>:
<meta name="robots" content="noindex">
You can combine it with nofollow if you also want links on the page not to be followed. For non-HTML files, use the equivalent X-Robots-Tag response header instead.
Keep it crawlable
The single most important rule: a crawler can only obey noindex if it can fetch the page. If you Disallow the URL in robots.txt, the crawler never reads the tag, and a page linked from elsewhere may stay in the index without a snippet.
So to deindex a page, leave it crawlable, serve noindex, and wait for the next recrawl. Only after it has dropped from search should you consider also disallowing it, if you no longer want it crawled at all.
- Page must remain crawlable for noindex to be read
- Do NOT also Disallow the URL in robots.txt
- Deindexing takes effect on the next recrawl
How it appears in analytics and logs
A noindex page should drop from search after it is recrawled. If it lingers, often the page was also disallowed, so the crawler never re-read the noindex signal.
Diagnostic use case
Keep a specific page (thank-you pages, thin pages, duplicates) out of search results while leaving it crawlable so the signal is read.
What WebmasterID can help detect
WebmasterID shows whether crawlers are still fetching the page, which is the prerequisite for your noindex to be seen and acted on.
Common mistakes
- Pairing noindex with a robots.txt Disallow, so the tag is never seen.
- Expecting instant removal — it happens on the next recrawl.
- Using noindex to try to hide private content from people (it does not).
Privacy and accuracy notes
noindex removes a page from search but does not hide it from people who have the link. Private content still needs authentication.
Related pages
- robots.txt vs the meta robots tag
robots.txt and the meta robots tag solve different problems. robots.txt asks crawlers not to fetch a path; the meta robots tag, embedded in a page's HTML, tells search engines whether to index it. The classic mistake is using Disallow to remove a page from search — which can backfire.
- robots.txt vs the X-Robots-Tag header
X-Robots-Tag carries the same indexing directives as the meta robots tag, but in the HTTP response header instead of the HTML body. That makes it the way to apply noindex or nofollow to non-HTML resources like PDFs and images, where a meta tag has nowhere to live.
- Website observability
Confirm a noindexed page is still being recrawled.
Sources and verification notes
- Google — Block search indexing with noindexDocuments noindex and the requirement that the page be crawlable.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.