Claude-SearchBot — Anthropic search crawler
Claude-SearchBot is a token Anthropic uses for crawling that supports Claude's search features. It is distinct from ClaudeBot (background crawler) and Claude-User (real-time user fetch). Anthropic documents its crawlers; where a specific is thin, it is marked partially verified rather than guessed.
What this means
Claude-SearchBot is the token Anthropic uses for crawling that supports Claude's search features. It sits alongside ClaudeBot, the background crawler, and Claude-User, the real-time user-triggered fetcher, each serving a distinct purpose.
Because these are separate tokens, a policy decision for one does not apply to the others. Where Anthropic's public documentation is thin on a particular specific, this entry describes the stable identification pattern and avoids asserting details that cannot be confidently sourced.
How Claude-SearchBot identifies itself
Claude-SearchBot uses the robots.txt user-agent token Claude-SearchBot. Its user-agent string contains that token together with a self-identifying URL. Match on the stable token rather than a full version string.
The user agent is a claim and can be copied. Use Anthropic's published guidance where authenticity matters, and do not invent IP ranges for verification.
- robots.txt token: Claude-SearchBot
- Supports Claude search features
- Distinct from ClaudeBot (training) and Claude-User (real-time fetch)
robots.txt considerations
To disallow Claude-SearchBot site-wide, target its token:
User-agent: Claude-SearchBot Disallow: /
This affects only the search crawler and does not block ClaudeBot or Claude-User. robots.txt is a request honoured by compliant crawlers, not an access-control boundary.
How it appears in analytics and logs
A request carrying the Claude-SearchBot token is Anthropic's search crawler fetching a URL — a bot event tied to search features, not background training and not a real-time user fetch. Identify it by the token and treat thin specifics conservatively.
Diagnostic use case
Identify Claude-SearchBot in logs by its token and set robots.txt policy for Anthropic's search crawler independently of ClaudeBot and Claude-User.
What WebmasterID can help detect
WebmasterID classifies Claude-SearchBot server-side by its token and surfaces its activity on the bot-intelligence and AI-visibility surfaces, so you can distinguish Anthropic's search crawl from ClaudeBot and Claude-User per page.
Common mistakes
- Assuming one Anthropic rule covers ClaudeBot, Claude-User, and Claude-SearchBot — each has its own token.
- Asserting documented behaviour where coverage is thin.
- Counting search-crawl hits as human sessions.
Privacy and accuracy notes
Detection uses only the request user-agent. No human identity is involved — a crawler is not a person. WebmasterID records the crawl as a bot event, separate from human analytics, and never attaches it to a visitor profile.
Related pages
- ClaudeBot — Anthropic's web crawler
ClaudeBot is the web crawler operated by Anthropic to fetch publicly available content. It is a declared crawler with a documented robots.txt token, and Anthropic publishes guidance for operators who want to identify or restrict it. It is separate from Claude-User, the agent that fetches pages when a person asks Claude to browse.
- Claude-User — Anthropic real-time fetcher
Claude-User is the token Anthropic uses for real-time fetches made when a person asks Claude to read a specific URL. It is distinct from ClaudeBot, the background crawler, and Anthropic documents both. It identifies itself with the Claude-User token plus a self-identifying URL.
- AI search analytics
Understand how AI search surfaces crawl and reference your site.
Sources and verification notes
- Anthropic — crawler and robots.txt guidanceDocuments Anthropic crawlers; some search-crawler specifics are thin, so marked partially verified.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.