Claude-User — Anthropic real-time fetcher
Claude-User is the token Anthropic uses for real-time fetches made when a person asks Claude to read a specific URL. It is distinct from ClaudeBot, the background crawler, and Anthropic documents both. It identifies itself with the Claude-User token plus a self-identifying URL.
What this means
Claude-User represents a real-time fetch Anthropic makes when a person asks Claude to read a specific URL. Unlike ClaudeBot, it is not a background crawl — it is triggered live by a user's request inside Claude.
Anthropic documents the two as separate tokens because they serve different purposes. Allowing or blocking ClaudeBot does not change Claude-User behaviour, and vice versa. Set each policy according to what you want to permit.
How Claude-User identifies itself
Claude-User uses the robots.txt user-agent token Claude-User. Its user-agent string contains that token together with a self-identifying URL. Match on the stable token rather than a full version string, because version components change over time.
The user agent is a claim and can be copied, so for requests that need to be trusted, use Anthropic's published verification guidance rather than the user agent alone.
- robots.txt token: Claude-User
- User agent contains the Claude-User token plus an Anthropic URL
- Distinct from ClaudeBot (background crawler)
robots.txt considerations
Claude-User honours robots.txt. To disallow it site-wide:
User-agent: Claude-User Disallow: /
This affects only the real-time fetcher and does not block ClaudeBot. robots.txt is a request honoured by compliant clients, not an access-control mechanism.
How it appears in analytics and logs
A request carrying the Claude-User token means Anthropic fetched the URL in real time because a Claude user asked it to read that page — a user-triggered bot fetch, not a background crawl and not a human page view. Treat it as automated traffic.
Diagnostic use case
Confirm whether a page was fetched on a Claude user's behalf, and set robots.txt policy for the real-time fetcher separately from the ClaudeBot crawler.
What WebmasterID can help detect
WebmasterID classifies Claude-User requests server-side as an AI fetcher and surfaces them on the bot-intelligence and AI-visibility surfaces, so you can see real-time Claude fetches separately from ClaudeBot crawls and from human visits.
Common mistakes
- Assuming a ClaudeBot block also stops Claude-User — they are separate tokens.
- Counting real-time Claude fetches as human page views.
- Treating robots.txt as enforcement rather than a request to compliant clients.
Privacy and accuracy notes
Detection uses only the request user-agent. The fetch is triggered by a person, but no visitor identity is exposed beyond a bot request. WebmasterID records it as a bot event, separate from human analytics, and never builds a visitor profile from it.
Related pages
- ClaudeBot — Anthropic's web crawler
ClaudeBot is the web crawler operated by Anthropic to fetch publicly available content. It is a declared crawler with a documented robots.txt token, and Anthropic publishes guidance for operators who want to identify or restrict it. It is separate from Claude-User, the agent that fetches pages when a person asks Claude to browse.
- GPTBot — OpenAI's web crawler
GPTBot is the crawler OpenAI uses to fetch publicly available web content that may be used to help train its foundation models. It is a declared, well-documented crawler with a stable robots.txt token, and OpenAI publishes both documentation and an IP range list so operators can identify and control it.
- AI referrals
Track human visits that arrive from AI assistants and answer engines.
Sources and verification notes
- Anthropic — crawler and robots.txt guidanceDocuments Anthropic's crawlers including the user-triggered fetcher.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.