Meta-ExternalAgent — Meta AI crawler
Meta-ExternalAgent is the token Meta uses for its crawler supporting AI products. Meta documents it alongside the related Meta-ExternalFetcher token. It identifies itself with the Meta-ExternalAgent token plus a self-identifying URL and honours robots.txt.
What this means
Meta-ExternalAgent is the crawler token Meta uses for fetching web content in support of its AI products. Meta documents it together with a related token, Meta-ExternalFetcher, which covers a different fetch behaviour.
Because the two tokens are distinct, a policy decision for one does not automatically apply to the other. Consult Meta's documentation to decide how to treat each token for your site.
How Meta-ExternalAgent identifies itself
Meta-ExternalAgent uses the robots.txt user-agent token Meta-ExternalAgent. Its user-agent string contains that token together with a self-identifying URL. Match on the stable token rather than a full version string.
The related token Meta-ExternalFetcher should be matched separately. As with any crawler, the user agent is a claim; use Meta's published guidance where authenticity matters, and do not invent IP ranges.
- robots.txt token: Meta-ExternalAgent
- Related token: Meta-ExternalFetcher (separate fetch behaviour)
- User agent contains the token plus a Meta URL
robots.txt considerations
Meta-ExternalAgent honours robots.txt. To disallow it site-wide:
User-agent: Meta-ExternalAgent Disallow: /
To also restrict the related fetcher, target Meta-ExternalFetcher separately. robots.txt is a request honoured by compliant crawlers, not an access-control boundary.
How it appears in analytics and logs
A request carrying the Meta-ExternalAgent token is Meta's AI crawler fetching a URL — a bot event, not a human visit. The related Meta-ExternalFetcher token represents a different fetch mode; identify each by its own token.
Diagnostic use case
Confirm whether Meta-ExternalAgent has crawled a page and set robots.txt policy for Meta's AI crawler, separately from its related fetcher token.
What WebmasterID can help detect
WebmasterID classifies Meta-ExternalAgent server-side as an AI crawler and surfaces its activity on the bot-intelligence and AI-visibility surfaces, so you can see Meta crawl coverage per page without parsing logs.
Common mistakes
- Assuming one rule covers both Meta-ExternalAgent and Meta-ExternalFetcher — they are separate tokens.
- Counting Meta crawler hits as human sessions.
- Expecting robots.txt to enforce access rather than request compliance.
Privacy and accuracy notes
Detection uses only the request user-agent. No human identity is involved — a crawler is not a person. WebmasterID records the crawl as a bot event, separate from human analytics, and never attaches it to a visitor profile.
Related pages
- GPTBot — OpenAI's web crawler
GPTBot is the crawler OpenAI uses to fetch publicly available web content that may be used to help train its foundation models. It is a declared, well-documented crawler with a stable robots.txt token, and OpenAI publishes both documentation and an IP range list so operators can identify and control it.
- Amazonbot — Amazon crawler
Amazonbot is the web crawler operated by Amazon. Amazon documents the crawler, its robots.txt token, and how site owners can control it. Amazonbot honours robots.txt and identifies itself with the Amazonbot token plus a self-identifying URL.
- AI visibility analytics
See which AI crawlers and assistants reach your site, recorded server-side.
Sources and verification notes
- Meta — crawler and robots.txt documentationDocuments Meta crawler tokens including Meta-ExternalAgent.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.