How to block ClaudeBot in robots.txt
If you do not want Anthropic's ClaudeBot fetching your site, you can disallow it in robots.txt. This page gives the exact rule, clarifies that it does not affect the real-time Claude-User fetch, and is honest that robots.txt is honoured by compliant crawlers rather than enforced.
The rule
Add this group to disallow ClaudeBot across the whole site:
User-agent: ClaudeBot Disallow: /
To block only part of the site, list specific paths instead of /, and use Allow to carve out exceptions.
What it does and does not affect
This rule targets only the ClaudeBot token, which Anthropic uses for its web crawler. It does not affect Claude-User, the separate token Anthropic uses when a person asks Claude to fetch a specific page in real time. If you also want to restrict that behaviour, target the Claude-User token in its own group.
The rule has no effect on crawlers from other companies — each is controlled by its own token.
- Affects: ClaudeBot (Anthropic's web crawler)
- Does NOT affect: Claude-User (real-time user-triggered fetch)
- Does NOT affect: other companies' AI crawlers
The limits
Anthropic documents that ClaudeBot honours robots.txt, but robots.txt cannot force compliance and a disallow does not retroactively remove content already used. Treat it as a forward-looking request to the compliant crawler.
How it appears in analytics and logs
After adding a ClaudeBot Disallow, compliant ClaudeBot requests to blocked paths should stop. Continued requests claiming to be ClaudeBot warrant verification.
Diagnostic use case
Disallow ClaudeBot site-wide or on specific paths while leaving Claude-User and other crawlers under their own rules.
What WebmasterID can help detect
WebmasterID shows ClaudeBot crawl activity before and after the change, so you can confirm the block took effect for the compliant crawler and flag any client ignoring it.
Common mistakes
- Expecting a ClaudeBot block to also stop the Claude-User real-time fetch.
- Typos in the token — it must be exactly ClaudeBot.
- Assuming robots.txt removes content already crawled in the past.
Privacy and accuracy notes
Blocking ClaudeBot is a publishing-policy choice. The rule is public in your robots.txt and is not a privacy or security mechanism.
Related pages
- How to block GPTBot in robots.txt
If you do not want OpenAI's training crawler fetching your site, you can disallow GPTBot in robots.txt. This page gives the exact rule, clarifies that it does not affect ChatGPT-User or OAI-SearchBot, and is honest about the limits of robots-based blocking.
- ClaudeBot — Anthropic's web crawler
ClaudeBot is the web crawler operated by Anthropic to fetch publicly available content. It is a declared crawler with a documented robots.txt token, and Anthropic publishes guidance for operators who want to identify or restrict it. It is separate from Claude-User, the agent that fetches pages when a person asks Claude to browse.
- AI visibility analytics
Confirm AI-crawler activity before and after a robots change.
Sources and verification notes
- Anthropic — crawler and robots.txt guidanceDocuments ClaudeBot and how site owners can block it.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.