How to block the CognitiveSEO crawler
CognitiveSEO is an SEO and backlink-analysis platform whose crawler fetches public pages to support its reports. This page shows the robots.txt token to target, what the crawler collects, and why a Disallow steers only compliant fetchers.
What the CognitiveSEO crawler is
CognitiveSEO is an SEO toolset that analyses backlinks, content, and rankings. Its crawler fetches public pages to support those analyses. Operators who do not want their site profiled by it can disallow the crawler.
Match on the documented CognitiveSEO user-agent token rather than a version string. The user agent is self-identifying; confirm the exact token from your logs.
- Purpose: SEO, content, and backlink analysis
- Operated by the CognitiveSEO platform
- Token is self-identifying; confirm it from your logs
robots.txt rule
To ask the CognitiveSEO crawler to stay off your site, target its documented token:
User-agent: CognitiveSEO Disallow: /
This targets only that token and leaves search and AI crawlers unaffected. robots.txt is honoured by compliant crawlers and is not enforcement; confirm the crawler backed off via crawl behavior.
How it appears in analytics and logs
Requests carrying the CognitiveSEO token are SEO/backlink-analysis crawl events, not human visits. They indicate a third-party platform is analysing your site; classify them as bot traffic.
Diagnostic use case
Keep an SEO and backlink-analysis crawler from fetching your pages for a third-party report and confirm the rule reached the correct token.
What WebmasterID can help detect
WebmasterID classifies the CognitiveSEO crawler server-side and shows whether it keeps reaching your pages after a robots.txt rule is added.
Common mistakes
- Guessing the token instead of confirming it from the self-identifying user agent.
- Expecting robots.txt to enforce the block rather than request compliance.
- Counting SEO-crawler hits as human sessions.
Privacy and accuracy notes
Blocking the CognitiveSEO crawler uses only the request user-agent token. No visitor identity is involved, and WebmasterID records the crawl as a bot event separate from human analytics.
Related pages
- CognitiveSEO crawler — backlink/SEO data bot
CognitiveSEO is an SEO and backlink-analysis platform whose crawler fetches pages to build link and on-page datasets for its subscribers. It is a third-party SEO tool crawler, not a search engine. This entry describes the documented pattern; confirm its current token in CognitiveSEO's materials before relying on it.
- How to block AhrefsBot in robots.txt
AhrefsBot is the crawler Ahrefs uses to build its backlink and SEO index. This page gives the robots.txt rule to disallow it and notes that Ahrefs documents support for both robots.txt rules and the crawl-delay directive, so you can slow rather than fully block it.
- How to block cocolyzebot
cocolyzebot is the crawler operated by Cocolyze, an SEO analytics platform that fetches pages to analyse rankings, backlinks, and on-page factors. This page shows the robots.txt token to target and why a Disallow steers only compliant fetchers.
- Bot intelligence
Categorise SEO and backlink crawlers in your traffic.
Sources and verification notes
- CognitiveSEO — crawler/bot informationCognitiveSEO crawler; token matched on the self-identifying user agent.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.