Claude-Web — legacy Anthropic token
Claude-Web is a historical robots.txt token associated with Anthropic's earlier crawling. Anthropic's current documented tokens are ClaudeBot and the user-triggered Claude-User. Where the legacy token's exact scope is unclear, it is marked partially verified rather than guessed.
What this means
Claude-Web is a historical robots.txt token associated with Anthropic's earlier crawling. Anthropic's current public guidance documents ClaudeBot for background crawling and Claude-User for real-time user-triggered fetches.
For robots.txt hygiene, a Claude-Web rule is the legacy name. To control Anthropic's present-day activity, target ClaudeBot and, if needed, Claude-User. Where the exact scope of the legacy token is unclear, this entry stays conservative rather than asserting specifics.
Migration to current tokens
If your robots.txt contains a legacy Claude-Web rule, the practical migration is to set rules on the current tokens that reflect your intended policy:
User-agent: ClaudeBot Disallow: / User-agent: Claude-User Disallow: /
Keeping the older Claude-Web rule alongside these is harmless. The active, documented controls for Anthropic today are ClaudeBot and Claude-User.
- Legacy token: Claude-Web
- Current tokens: ClaudeBot and Claude-User
- An unused legacy rule is harmless to keep
How it appears in analytics and logs
Seeing Claude-Web in older robots.txt files reflects an earlier Anthropic token. Current Anthropic crawling is documented under ClaudeBot and Claude-User, so treat Claude-Web as historical and verify present-day activity against the current tokens.
Diagnostic use case
Understand legacy Claude-Web robots.txt rules and migrate to controlling the current ClaudeBot and Claude-User tokens.
What WebmasterID can help detect
WebmasterID classifies current Anthropic activity under ClaudeBot and Claude-User. The legacy Claude-Web token is historical; WebmasterID surfaces the crawlers it actually observes rather than deprecated names.
Common mistakes
- Assuming a Claude-Web rule still governs current Anthropic activity — ClaudeBot and Claude-User are the active tokens.
- Confusing the legacy Claude-Web token with the current Claude-User real-time fetcher.
- Inventing details about the legacy token's exact scope.
Privacy and accuracy notes
This entry concerns robots.txt tokens, not visitor data. A legacy token in your robots.txt is a policy artefact and involves no human identity. WebmasterID records any current crawl as a bot event only.
Related pages
- anthropic-ai — Anthropic legacy token
anthropic-ai is a historical robots.txt token associated with Anthropic's earlier crawling, now superseded by ClaudeBot. Anthropic's current guidance centres on ClaudeBot and its user-triggered fetcher. Keeping a legacy rule is harmless but the active control is ClaudeBot.
- Claude-User — Anthropic real-time fetcher
Claude-User is the token Anthropic uses for real-time fetches made when a person asks Claude to read a specific URL. It is distinct from ClaudeBot, the background crawler, and Anthropic documents both. It identifies itself with the Claude-User token plus a self-identifying URL.
- Web crawlers reference
Reference for crawlers, control tokens, and how they appear in traffic.
Sources and verification notes
- Anthropic — crawler and robots.txt guidanceCurrent guidance documents ClaudeBot and Claude-User; the legacy token's exact scope is not exhaustively documented.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.