How to block cohere-ai in robots.txt
cohere-ai is the robots.txt token associated with Cohere's web fetching for its AI products. This page gives the rule to disallow the cohere-ai token, explains where it fits in an AI-crawler policy, and stays cautious where Cohere's public documentation is limited.
What cohere-ai is
cohere-ai is the robots.txt token associated with Cohere, a company that builds large language models and AI products for businesses. Operators who want to keep their content out of Cohere's web fetching add this token to an AI-crawler opt-out alongside tokens for other AI crawlers.
Cohere's public crawler documentation is limited compared with some larger providers, so this entry treats specifics cautiously and focuses on the observed token rather than asserting a full user-agent string or IP ranges.
The rule
To disallow cohere-ai site-wide, target the token:
User-agent: cohere-ai Disallow: /
Match the token exactly. Because cohere-ai is one of many AI tokens, blocking it does not affect other AI crawlers — pair it with a wider AI-crawler policy if your goal is a broad opt-out. robots.txt is honoured by compliant crawlers and is not enforcement.
- Token: cohere-ai
- Associated with Cohere's AI products
- One of many AI tokens — pair with a broader policy
How it appears in analytics and logs
A request carrying the cohere-ai token is web fetching associated with Cohere's AI products reaching a URL. It is a bot event, not a human visit, and brings no audience by itself.
Diagnostic use case
Disallow cohere-ai when you want to limit Cohere's AI-related web fetching of your content, as part of a broader opt-out from AI crawling.
What WebmasterID can help detect
WebmasterID classifies cohere-ai by its token as an AI crawler, separate from human analytics, so you can see whether a disallow rule changed its activity.
Common mistakes
- Expecting cohere-ai to cover all AI crawlers — it is one token among many.
- Inventing a user-agent string or IP ranges where Cohere's docs are sparse.
- Mistyping the token — it must be exactly cohere-ai.
Privacy and accuracy notes
Blocking cohere-ai is a publishing-policy choice in a public file. It involves no visitor data and is not an access-control mechanism.
Related pages
- cohere-ai — Cohere crawler
cohere-ai is a crawler token associated with Cohere. It appears in server logs as an automated fetcher. Public documentation is limited, so specifics about its purpose and behaviour are marked partially verified rather than guessed; no behaviour is invented.
- Writing an AI crawler policy for robots.txt
An AI crawler policy is a deliberate decision about which AI-related tokens you allow and which you disallow in robots.txt. This page offers a structured way to make and document those choices, while staying realistic: robots.txt is a request to compliant crawlers, not a legal or technical guarantee.
- How to block omgilibot in robots.txt
omgilibot is the crawler historically associated with Omgili and the Webz.io web-data project, which collects public web content for datasets resold to third parties, including AI uses. This page gives the robots.txt rule to disallow the omgilibot token and flags where documentation is limited.
- AI visibility analytics
See which AI crawlers reach your site, recorded server-side.
Sources and verification notes
- Cohere — crawler token reference (observed)Token cohere-ai is observed in AI opt-out contexts; comprehensive official docs are limited, so specifics are marked partially verified.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.