How to block all bots in robots.txt
A single robots.txt group can ask every compliant crawler to stay off your whole site. This page gives the exact rule and is blunt about the caveats: robots.txt is advisory rather than enforced, blocking search crawlers can remove you from results, and it is not a security boundary.
The rule
To ask every compliant crawler to avoid the entire site, use the default group with a root Disallow:
User-agent: * Disallow: /
The * matches any crawler that does not have its own more specific group, and Disallow: / covers every path.
The caveats
This rule is widely misunderstood. It is advisory — compliant crawlers honour it, but nothing enforces it, so it is not a way to protect content. More importantly, blocking search crawlers like Googlebot and Bingbot can remove your pages from search results over time, because they can no longer recrawl them.
It is also not the way to deindex a single page: if a blocked URL is linked elsewhere it can still appear in results without a snippet. To keep a page out of search, allow crawling and use a noindex signal instead.
- Advisory, not enforcement — non-compliant clients ignore it
- Can deindex you from search if you block search crawlers
- Not a substitute for noindex when you want to leave search
How it appears in analytics and logs
A block-all robots.txt means compliant crawlers should request nothing. If you still see crawler hits on disallowed paths, those clients are ignoring robots.txt and warrant closer inspection.
Diagnostic use case
Discourage all compliant crawling site-wide (for example on a staging host) while understanding that this does not stop non-compliant clients.
What WebmasterID can help detect
WebmasterID lets you see which crawlers continue to hit blocked paths after a block-all rule, helping you distinguish compliant crawlers from clients that disregard the standard.
Common mistakes
- Using Disallow: / on a production site and losing search visibility.
- Expecting a block-all rule to stop scrapers — it cannot enforce anything.
- Trying to deindex one page with Disallow instead of noindex.
Privacy and accuracy notes
Blocking all bots is not access control. The file is public, and non-compliant clients can ignore it, so private content still needs authentication.
Related pages
- How to allow all bots in robots.txt
If you want every compliant crawler to be free to fetch your whole site, robots.txt makes that the default. This page shows the explicit allow-all group, explains why an empty or absent file is also open, and clarifies that allowing crawling is not the same as forcing indexing.
- The noindex meta tag
The noindex value of the meta robots tag tells search engines to keep a page out of their index. The catch trips people up constantly: for noindex to work, the crawler must be able to fetch the page — so you must not block the same URL in robots.txt.
- Web crawlers
Reference for the crawlers a block-all rule addresses.
Sources and verification notes
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.