How to block SemrushBot in robots.txt
SemrushBot is the crawler Semrush uses to build its SEO datasets. Semrush documents several specialised sub-bots under related tokens, so this page covers the base disallow rule and explains why you may need to target multiple tokens to cover the activity you care about.
What SemrushBot is
SemrushBot crawls the web for the datasets behind the Semrush platform. It is a third-party SEO crawler with no effect on Google or Bing rankings and no role in AI training. Operators sometimes block it to cut crawl load from tools they do not subscribe to.
The rule and sub-bots
To disallow the main crawler site-wide:
User-agent: SemrushBot Disallow: /
Semrush documents a family of specialised crawlers — for example separate tokens used for backlink, site-audit, and other features. Because robots.txt matching is per-token, a rule for SemrushBot may not cover every Semrush sub-bot. Check Semrush's bot documentation for the current list and add a group for each token you want to disallow.
- Base token: SemrushBot
- Multiple specialised sub-bot tokens exist
- robots.txt matches per token — add a group per token
How it appears in analytics and logs
A request with a SemrushBot token is Semrush building its index. It is third-party SEO crawling — not a search engine, not a human visit. Different Semrush features may appear under different sub-bot tokens.
Diagnostic use case
Disallow SemrushBot and its relevant sub-bots to reduce SEO-tool crawl load, without affecting search engines or AI crawlers.
What WebmasterID can help detect
WebmasterID classifies Semrush crawlers as SEO crawlers separate from human traffic, so you can confirm which tokens still appear after a robots.txt change.
Common mistakes
- Blocking SemrushBot and assuming every Semrush sub-bot is covered.
- Expecting a SemrushBot block to change Google or Bing results.
- Misspelling tokens — each must match exactly.
Privacy and accuracy notes
Managing SemrushBot is a crawl-load and policy choice expressed in a public file. It involves no visitor data.
Related pages
- How to block AhrefsBot in robots.txt
AhrefsBot is the crawler Ahrefs uses to build its backlink and SEO index. This page gives the robots.txt rule to disallow it and notes that Ahrefs documents support for both robots.txt rules and the crawl-delay directive, so you can slow rather than fully block it.
- User-agent groups and matching in robots.txt
robots.txt rules are organised into user-agent groups. A crawler does not combine every group — it selects the single most specific group whose token matches its name, falling back to the * group only when no named group matches. Understanding this prevents rules that never apply.
- Bot intelligence
See which SEO-crawler tokens still reach your site.
Sources and verification notes
- Semrush — SemrushBot documentationDocuments SemrushBot and its specialised sub-bots.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.