AI2Bot — Allen Institute for AI crawler
AI2Bot is the crawler operated by the Allen Institute for AI (AI2) to gather web data for its datasets and research. AI2 documents the crawler and its robots.txt token. Where a specific is not clearly covered it is marked partially verified rather than guessed.
What this means
AI2Bot is the crawler the Allen Institute for AI uses to gather public web data for its datasets and research. It appears in logs as an automated fetcher carrying the AI2Bot token.
AI2 publishes guidance on the crawler. Where a particular specific is not clearly covered, this entry describes the stable identification pattern rather than asserting unsourced detail.
How AI2Bot identifies itself
AI2Bot uses the robots.txt user-agent token AI2Bot. Its user-agent string contains that token together with a self-identifying URL. Match on the stable token rather than a full version string.
The user agent is a claim and can be copied. Use AI2's published guidance where authenticity matters, and do not invent IP ranges.
- robots.txt token: AI2Bot
- Operated by the Allen Institute for AI (AI2)
- Gathers web data for datasets and research
robots.txt considerations
To disallow AI2Bot site-wide:
User-agent: AI2Bot Disallow: /
AI2Bot is expected to honour robots.txt as a compliant crawler. robots.txt is a request, not an access-control boundary.
How it appears in analytics and logs
A request carrying the AI2Bot token is the Allen Institute for AI's crawler fetching a URL for its datasets — a bot event, not a human visit. Identify it by the token and treat any unverified specifics conservatively.
Diagnostic use case
Identify AI2Bot in logs by its token and set robots.txt policy for the Allen Institute for AI's dataset crawler.
What WebmasterID can help detect
WebmasterID classifies AI2Bot server-side by its token and surfaces it on the bot-intelligence and AI-visibility surfaces, so you can see Allen Institute crawl activity per page without parsing logs.
Common mistakes
- Asserting detailed behaviour where coverage is partial.
- Inventing IP ranges to verify AI2Bot.
- Counting crawler hits as human sessions.
Privacy and accuracy notes
Detection uses only the request user-agent. No human identity is involved. WebmasterID records the crawl as a bot event, separate from human analytics, and never attaches it to a visitor profile.
Related pages
- CCBot — Common Crawl crawler
CCBot is the crawler operated by Common Crawl to build its open, freely available web dataset. That dataset is widely reused as a training source by many AI projects. Common Crawl documents the crawler and its robots.txt token, and CCBot honours robots.txt.
- cohere-ai — Cohere crawler
cohere-ai is a crawler token associated with Cohere. It appears in server logs as an automated fetcher. Public documentation is limited, so specifics about its purpose and behaviour are marked partially verified rather than guessed; no behaviour is invented.
- Bot intelligence
Deterministic categorisation of crawlers, search bots, and automation.
Sources and verification notes
- Allen Institute for AI — AI2Bot crawler guidanceDocuments AI2Bot; some specifics marked partially verified.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.