Meta-ExternalFetcher — Meta on-demand fetch
Meta-ExternalFetcher is the token Meta uses for on-demand fetches, as opposed to Meta-ExternalAgent, its bulk AI crawler. Meta documents both. It identifies itself with the Meta-ExternalFetcher token plus a self-identifying URL and is controlled separately in robots.txt.
What this means
Meta-ExternalFetcher is the token Meta uses for on-demand fetches, distinct from Meta-ExternalAgent, which is its bulk crawler supporting AI products. The on-demand fetcher retrieves a specific resource when needed rather than crawling broadly.
Meta documents the two tokens separately because they behave differently. A policy decision for Meta-ExternalAgent does not automatically apply to Meta-ExternalFetcher, so set each according to what you want to permit.
How Meta-ExternalFetcher identifies itself
Meta-ExternalFetcher uses the robots.txt user-agent token Meta-ExternalFetcher. Its user-agent string contains that token together with a self-identifying URL. Match on the stable token rather than a full version string.
The user agent is a claim and can be copied. Use Meta's published guidance where authenticity matters, and do not invent IP ranges.
- robots.txt token: Meta-ExternalFetcher
- On-demand fetch, distinct from Meta-ExternalAgent (bulk crawl)
- User agent contains the token plus a Meta URL
robots.txt considerations
To disallow Meta-ExternalFetcher site-wide, target its token:
User-agent: Meta-ExternalFetcher Disallow: /
This affects only the on-demand fetcher and does not block Meta-ExternalAgent. robots.txt is a request honoured by compliant crawlers, not an access-control boundary.
How it appears in analytics and logs
A request carrying the Meta-ExternalFetcher token is an on-demand fetch by Meta — a bot event tied to a specific need, not the bulk Meta-ExternalAgent crawl and not a human visit. Identify each Meta token separately.
Diagnostic use case
Confirm whether Meta-ExternalFetcher fetched a page on demand and set robots.txt policy for it independently of the Meta-ExternalAgent crawler.
What WebmasterID can help detect
WebmasterID classifies Meta-ExternalFetcher server-side by its token and surfaces it on the bot-intelligence and AI-visibility surfaces, so you can distinguish Meta's on-demand fetch from its Meta-ExternalAgent bulk crawl per page.
Common mistakes
- Assuming one rule covers both Meta-ExternalFetcher and Meta-ExternalAgent — they are separate tokens.
- Counting on-demand fetches as human page views.
- Expecting robots.txt to enforce access rather than request compliance.
Privacy and accuracy notes
Detection uses only the request user-agent. No human identity is involved — a crawler is not a person. WebmasterID records the crawl as a bot event, separate from human analytics, and never attaches it to a visitor profile.
Related pages
- Meta-ExternalAgent — Meta AI crawler
Meta-ExternalAgent is the token Meta uses for its crawler supporting AI products. Meta documents it alongside the related Meta-ExternalFetcher token. It identifies itself with the Meta-ExternalAgent token plus a self-identifying URL and honours robots.txt.
- Real-time AI fetcher agents
Real-time AI fetcher agents — such as ChatGPT-User, Claude-User, and Perplexity-User — retrieve a specific page live when a person asks an assistant about it. They are user-triggered, not bulk crawls, and each has its own robots.txt token controlled separately from the vendor's background crawler.
- Bot intelligence
Deterministic categorisation of crawlers, search bots, and automation.
Sources and verification notes
- Meta — crawler and robots.txt documentationDocuments Meta crawler tokens including Meta-ExternalFetcher and Meta-ExternalAgent.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.