AI assistant user agents
Separate from background training crawlers, AI assistants make real-time fetches when a person asks them to read or browse a page. These use distinct tokens — ChatGPT-User, Claude-User, Perplexity-User — each with a self-identifying URL. This page explains the assistant-fetch pattern and links to the AI-crawlers hub.
The assistant-fetch pattern
When a person asks an AI assistant to read or browse a specific page, the assistant fetches it in real time. These fetches use their own tokens — ChatGPT-User for OpenAI, Claude-User for Anthropic, Perplexity-User for Perplexity — each accompanied by a self-identifying URL pointing at the operator's documentation.
This is deliberately separate from the background training crawlers (GPTBot, ClaudeBot, PerplexityBot). The assistant token signals a live, user-prompted fetch rather than a bulk crawl.
- ChatGPT-User, Claude-User, Perplexity-User real-time tokens
- Triggered live by a person's request, not bulk crawling
- Distinct from training crawlers GPTBot, ClaudeBot, PerplexityBot
Reading and controlling assistant fetches
Because assistant fetches and training crawls use different tokens, a robots.txt policy for one does not carry to the other. If you want assistants to be able to read pages on a user's behalf, allow the assistant token; otherwise disallow it specifically.
For per-token details — exact identifiers, robots.txt controls, and verification — see the individual entries in the AI-crawlers hub. Match on the stable token, not a full version string, and treat the user agent as a claim to be verified where it matters.
How it appears in analytics and logs
A user agent carrying a real-time assistant token (ChatGPT-User, Claude-User, Perplexity-User) means an assistant fetched the page because a person asked about it. It is user-triggered automation, not a background crawl and not a human page view.
Diagnostic use case
Recognise user-triggered AI assistant fetches by their tokens, distinguish them from background AI training crawlers, and find per-token controls in the AI-crawlers hub.
What WebmasterID can help detect
WebmasterID classifies real-time assistant fetchers server-side and surfaces them on the bot-intelligence and AI-visibility surfaces, distinct from background AI crawlers and from human visits.
Common mistakes
- Conflating real-time assistant fetches with background AI training crawls.
- Counting assistant fetches as human page views or audience growth.
- Assuming one robots.txt rule covers both the assistant token and the training token.
Privacy and accuracy notes
These fetches are prompted by a person, but no visitor identity reaches your server beyond a bot request. WebmasterID records them as bot events, separate from human analytics, and never builds a visitor profile.
Related pages
- ChatGPT-User — OpenAI real-time fetcher
ChatGPT-User is the token OpenAI uses for real-time fetches made when a person in ChatGPT browses or asks it to read a URL. It is distinct from GPTBot, which crawls for model training, and OpenAI documents both. It honours robots.txt and identifies itself with the ChatGPT-User token plus a self-identifying URL.
- Claude-User — Anthropic real-time fetcher
Claude-User is the token Anthropic uses for real-time fetches made when a person asks Claude to read a specific URL. It is distinct from ClaudeBot, the background crawler, and Anthropic documents both. It identifies itself with the Claude-User token plus a self-identifying URL.
- AI crawler user agents
AI crawlers from companies building and serving large models fetch public web content. Their user agents follow a recognisable shape: a product token plus a self-identifying URL pointing at the operator's documentation. This page explains how to read the AI-crawler pattern and links to the AI-crawlers hub for specifics.
- AI referrals
Track human visits that arrive from AI assistants and answer engines.
Sources and verification notes
- OpenAI — bots documentationDocuments ChatGPT-User as distinct from GPTBot.
- Perplexity — crawler documentationDocuments Perplexity-User real-time fetches.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.