SEO crawler user agents
SEO platforms run their own crawlers to build backlink indexes and audit data. Bots such as AhrefsBot, SemrushBot, and DotBot identify themselves with a documented token and honour robots.txt. They are not search-engine indexers. This page explains the family and how to recognise and control it.
What SEO crawlers are for
SEO platforms operate crawlers to discover links, audit pages, and build the datasets behind their backlink and site-audit tools. AhrefsBot, SemrushBot, and DotBot are well-known examples. They crawl the web to populate a commercial index, distinct from a search engine's own indexing crawler.
They are related to, but not the same as, the search-engine bots covered in the search-bots category; SEO crawlers do not place you in Google or Bing results.
- AhrefsBot, SemrushBot, DotBot and similar tool crawlers
- Build backlink and audit datasets, not search results
- Each identifies with a documented token and honours robots.txt
Recognising and controlling them
Each SEO crawler publishes its robots.txt token and documentation. Compliant ones honour robots.txt, so you can allow or disallow a specific token. Allowing them means your site appears in that tool's data; disallowing asks the crawler to stay out.
Match on the documented token rather than a full version string, and verify a claimed SEO crawler the same way you verify any crawler where authenticity matters, since the string can be spoofed.
How it appears in analytics and logs
A request from an SEO-crawler token (AhrefsBot, SemrushBot, DotBot, and similar) is a backlink or audit crawl, not a human visit and not Google indexing. Sustained activity reflects crawl coverage by that tool.
Diagnostic use case
Recognise SEO and backlink crawlers, count them as automation, and decide whether to allow them in robots.txt based on whether you want to appear in their tools.
What WebmasterID can help detect
WebmasterID classifies common SEO crawlers server-side as SEO/automation bots, separate from search bots and human analytics, so their crawl activity is visible without log parsing.
Common mistakes
- Confusing SEO crawlers with search-engine indexers like Googlebot.
- Counting SEO crawl hits as human traffic.
- Trusting an SEO-crawler user agent without verification when it matters.
Privacy and accuracy notes
SEO crawlers carry no visitor identity. WebmasterID records their requests as bot events, separate from human analytics, never as profiles.
Related pages
- AI crawler user agents
AI crawlers from companies building and serving large models fetch public web content. Their user agents follow a recognisable shape: a product token plus a self-identifying URL pointing at the operator's documentation. This page explains how to read the AI-crawler pattern and links to the AI-crawlers hub for specifics.
- Bot vs browser user agents: how to tell them apart
A user-agent string is a self-reported label, not an identity. This page explains how declared bots name themselves, why almost every UA still starts with the legacy Mozilla token, and how to read the difference between an automated client and a real browser without over-trusting the string.
- Web crawlers
Reference for the crawlers that visit your site and how to control them.
Sources and verification notes
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.