How to block Meta-ExternalAgent in robots.txt
Meta-ExternalAgent is the token Meta uses for its crawler supporting AI products. This page gives the robots.txt rule to disallow it and explains that the related Meta-ExternalFetcher token covers a different fetch behaviour and must be targeted separately.
What Meta-ExternalAgent is
Meta-ExternalAgent is the crawler token Meta uses to fetch web content in support of its AI products. Meta documents it together with a related token, Meta-ExternalFetcher, which covers a different fetch behaviour. Because the two are distinct tokens, a rule for one does not automatically cover the other.
The rule and the related token
To disallow the AI crawler site-wide:
User-agent: Meta-ExternalAgent Disallow: /
To also restrict the related fetcher, add a separate group:
User-agent: Meta-ExternalFetcher Disallow: /
robots.txt matches per token, so each token you want to cover needs its own group. robots.txt is honoured by compliant crawlers, not enforced, and is not an access-control boundary.
- Token: Meta-ExternalAgent
- Related token: Meta-ExternalFetcher (target separately)
- Per-token matching — add a group for each token
How it appears in analytics and logs
After adding a Meta-ExternalAgent Disallow, compliant requests under that token to blocked paths should stop. Meta-ExternalFetcher activity continues unless you target it too.
Diagnostic use case
Disallow Meta-ExternalAgent site-wide, and decide separately whether to also restrict the related Meta-ExternalFetcher token.
What WebmasterID can help detect
WebmasterID shows Meta crawler activity per token, so you can confirm which Meta tokens still appear after a robots.txt change.
Common mistakes
- Assuming a Meta-ExternalAgent rule also covers Meta-ExternalFetcher — they are separate tokens.
- Misspelling the tokens — each must match exactly.
- Expecting robots.txt to enforce access rather than request compliance.
Privacy and accuracy notes
Blocking Meta-ExternalAgent is a publishing-policy choice in a public file. It involves no visitor data.
Related pages
- Meta-ExternalAgent — Meta AI crawler
Meta-ExternalAgent is the token Meta uses for its crawler supporting AI products. Meta documents it alongside the related Meta-ExternalFetcher token. It identifies itself with the Meta-ExternalAgent token plus a self-identifying URL and honours robots.txt.
- User-agent groups and matching in robots.txt
robots.txt rules are organised into user-agent groups. A crawler does not combine every group — it selects the single most specific group whose token matches its name, falling back to the * group only when no named group matches. Understanding this prevents rules that never apply.
- Writing an AI crawler policy for robots.txt
An AI crawler policy is a deliberate decision about which AI-related tokens you allow and which you disallow in robots.txt. This page offers a structured way to make and document those choices, while staying realistic: robots.txt is a request to compliant crawlers, not a legal or technical guarantee.
- AI visibility analytics
See which Meta tokens still reach your site.
Sources and verification notes
- Meta — crawler and robots.txt documentationDocuments Meta crawler tokens including Meta-ExternalAgent.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.