cohere-ai — Cohere crawler
cohere-ai is a crawler token associated with Cohere. It appears in server logs as an automated fetcher. Public documentation is limited, so specifics about its purpose and behaviour are marked partially verified rather than guessed; no behaviour is invented.
What this means
cohere-ai is a crawler token associated with Cohere. It appears in logs as an automated fetcher carrying the cohere-ai token.
Public documentation about this crawler is limited. For that reason, this entry describes the stable identification pattern and avoids asserting specifics — such as exact crawl purpose or scope — that cannot be confidently sourced. No behaviour is invented here.
How cohere-ai identifies itself
The crawler uses the robots.txt user-agent token cohere-ai. Its user-agent string contains that token. Match on the stable token rather than a full version string.
The user agent is a claim and can be copied. Because verification guidance is not clearly published, do not invent IP ranges; identify the crawler by its token and treat trust-sensitive decisions conservatively.
- robots.txt token: cohere-ai
- Associated with Cohere
- Specifics beyond the token: not fully documented publicly
robots.txt considerations
To disallow cohere-ai site-wide, target its token:
User-agent: cohere-ai Disallow: /
Whether and how strictly it honours robots.txt is not clearly documented, so treat this rule as a request rather than a guarantee. robots.txt is never an access-control boundary.
How it appears in analytics and logs
A request carrying the cohere-ai token is a Cohere-associated crawler fetching a URL — a bot event, not a human visit. Identify it by the token; because public docs are limited, treat its exact purpose conservatively.
Diagnostic use case
Identify cohere-ai activity in logs by its token and set robots.txt policy, treating undocumented specifics with caution.
What WebmasterID can help detect
WebmasterID classifies cohere-ai server-side by its token and surfaces it on the bot-intelligence surface, so you can see its activity per page without parsing logs.
Common mistakes
- Asserting documented behaviour where public docs are sparse.
- Inventing IP ranges to verify cohere-ai.
- Counting crawler hits as human sessions.
Privacy and accuracy notes
Detection uses only the request user-agent. No human identity is involved. WebmasterID records the crawl as a bot event, separate from human analytics, and never attaches it to a visitor profile.
Related pages
- AI2Bot — Allen Institute for AI crawler
AI2Bot is the crawler operated by the Allen Institute for AI (AI2) to gather web data for its datasets and research. AI2 documents the crawler and its robots.txt token. Where a specific is not clearly covered it is marked partially verified rather than guessed.
- YouBot — You.com crawler
YouBot is the crawler operated by You.com to support its search and AI assistant. Its robots.txt token is YouBot. Public documentation is limited in places, so specifics that cannot be confidently sourced are marked partially verified rather than guessed.
- Bot intelligence
Deterministic categorisation of crawlers, search bots, and automation.
Sources and verification notes
- Cohere — crawler reference (token observed)Token cohere-ai is observed; comprehensive official docs are limited, so some specifics are marked partially verified.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.