Real-time AI fetcher agents
Real-time AI fetcher agents — such as ChatGPT-User, Claude-User, and Perplexity-User — retrieve a specific page live when a person asks an assistant about it. They are user-triggered, not bulk crawls, and each has its own robots.txt token controlled separately from the vendor's background crawler.
What a real-time fetcher is
A real-time AI fetcher retrieves a single page live, in response to a user's request inside an assistant. When someone asks ChatGPT, Claude, or Perplexity about a specific URL or a question that requires reading one, the assistant fetches that page on demand. The tokens for this behaviour include ChatGPT-User, Claude-User, and Perplexity-User.
This is fundamentally different from a bulk crawl. A background crawler systematically visits many pages on a schedule; a real-time fetcher visits one page because a person needed it right then.
Controlling the fetcher family
Each real-time fetcher has its own robots.txt token, controlled separately from the vendor's background crawler. Blocking GPTBot does not stop ChatGPT-User; blocking ClaudeBot does not stop Claude-User; blocking PerplexityBot does not stop Perplexity-User.
Decide your policy per token. If you want assistants to be able to read your pages when users ask, allow the fetcher tokens; if not, disallow them specifically. As always, robots.txt is a request honoured by compliant clients, not an access-control boundary.
- Examples: ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User
- User-triggered, not scheduled bulk crawling
- Each controlled by its own robots.txt token
How it appears in analytics and logs
A real-time fetcher token in your logs means an assistant fetched that page because a user asked, not because a scheduled crawl reached it. Volume tends to track user interest in specific pages rather than systematic coverage.
Diagnostic use case
Recognise user-triggered AI fetches in your logs and control them with their own robots.txt tokens, separate from background crawlers.
What WebmasterID can help detect
WebmasterID classifies real-time fetcher agents as a distinct family server-side, so ChatGPT-User, Claude-User, Perplexity-User and similar tokens appear separately from bulk crawlers on the bot-intelligence surface.
Common mistakes
- Assuming a background-crawler block also stops the matching real-time fetcher.
- Counting user-triggered fetches as human page views.
- Treating fetch volume as systematic crawl coverage rather than user-driven interest.
Privacy and accuracy notes
These fetches are triggered by people, but no visitor identity is exposed to your server beyond a bot request. WebmasterID records each as a bot event, separate from human analytics, and never builds a visitor profile from it.
Related pages
- ChatGPT-User — OpenAI real-time fetcher
ChatGPT-User is the token OpenAI uses for real-time fetches made when a person in ChatGPT browses or asks it to read a URL. It is distinct from GPTBot, which crawls for model training, and OpenAI documents both. It honours robots.txt and identifies itself with the ChatGPT-User token plus a self-identifying URL.
- Claude-User — Anthropic real-time fetcher
Claude-User is the token Anthropic uses for real-time fetches made when a person asks Claude to read a specific URL. It is distinct from ClaudeBot, the background crawler, and Anthropic documents both. It identifies itself with the Claude-User token plus a self-identifying URL.
- AI referrals
Track human visits that arrive from AI assistants and answer engines.
Sources and verification notes
- OpenAI — bots documentationDocuments ChatGPT-User as a user-triggered fetcher distinct from GPTBot.
- Anthropic — crawler and robots.txt guidanceDocuments Claude-User as a user-triggered fetch distinct from ClaudeBot.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.