anthropic-ai — Anthropic legacy token
anthropic-ai is a historical robots.txt token associated with Anthropic's earlier crawling, now superseded by ClaudeBot. Anthropic's current guidance centres on ClaudeBot and its user-triggered fetcher. Keeping a legacy rule is harmless but the active control is ClaudeBot.
What this means
anthropic-ai is a historical robots.txt token associated with Anthropic's earlier crawling activity. Anthropic's current public guidance centres on ClaudeBot for background crawling, plus user-triggered fetch tokens such as Claude-User.
This matters for robots.txt hygiene: a rule targeting anthropic-ai is not wrong to keep, but it is the legacy name. To control Anthropic's present-day crawling, target ClaudeBot. Where the exact history of the token is not fully documented, this entry stays conservative rather than guessing.
Migration to ClaudeBot
If your robots.txt contains a legacy anthropic-ai rule, the practical migration is to add or maintain a ClaudeBot rule that reflects your intended policy:
User-agent: ClaudeBot Disallow: /
You can keep the older anthropic-ai rule alongside it; an unused token rule is harmless. The active, documented control for Anthropic crawling today is ClaudeBot, with separate tokens for user-triggered fetches.
- Legacy token: anthropic-ai
- Current crawler token: ClaudeBot
- Keeping a legacy rule is harmless; the active control is ClaudeBot
How it appears in analytics and logs
Seeing anthropic-ai in older robots.txt files reflects an earlier Anthropic token. Current Anthropic crawling is documented under ClaudeBot, so treat anthropic-ai as historical and verify present-day activity against the ClaudeBot token.
Diagnostic use case
Understand legacy anthropic-ai robots.txt rules and migrate to controlling ClaudeBot, which is Anthropic's current crawler token.
What WebmasterID can help detect
WebmasterID classifies current Anthropic crawling under ClaudeBot. The legacy anthropic-ai token is historical; WebmasterID surfaces the crawlers it actually observes rather than deprecated token names.
Common mistakes
- Assuming an anthropic-ai rule still governs current Anthropic crawling — ClaudeBot is the active token.
- Removing all Anthropic rules during cleanup and unintentionally changing your policy.
- Inventing details about the legacy token's exact history.
Privacy and accuracy notes
This entry concerns robots.txt tokens, not visitor data. A legacy token in your robots.txt is a policy artefact and involves no human identity. WebmasterID records any current crawl as a bot event only.
Related pages
- ClaudeBot — Anthropic's web crawler
ClaudeBot is the web crawler operated by Anthropic to fetch publicly available content. It is a declared crawler with a documented robots.txt token, and Anthropic publishes guidance for operators who want to identify or restrict it. It is separate from Claude-User, the agent that fetches pages when a person asks Claude to browse.
- Claude-Web — legacy Anthropic token
Claude-Web is a historical robots.txt token associated with Anthropic's earlier crawling. Anthropic's current documented tokens are ClaudeBot and the user-triggered Claude-User. Where the legacy token's exact scope is unclear, it is marked partially verified rather than guessed.
- Web crawlers reference
Reference for crawlers, control tokens, and how they appear in traffic.
Sources and verification notes
- Anthropic — crawler and robots.txt guidanceCurrent guidance centres on ClaudeBot; the legacy token's full history is not exhaustively documented.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.