SemrushBot — Semrush SEO crawler
SemrushBot is the crawler operated by Semrush to gather data for its SEO toolset. It is a third-party crawler, not a search engine, so it does not affect search rankings directly. It uses the SemrushBot robots.txt token and is documented as respecting robots.txt.
What this means
SemrushBot crawls the web to gather data for the Semrush SEO and competitive-analysis toolset. It is a third-party crawler, distinct from search-engine crawlers, and its visits do not directly change your rankings on Google or Bing.
Semrush documents multiple SemrushBot variants for different features; they share the SemrushBot user-agent identifier.
robots.txt considerations
Semrush documents that SemrushBot respects robots.txt. To exclude it, target the SemrushBot token with a Disallow rule; consult Semrush's documentation for variant-specific tokens and supported directives. The user agent is spoofable, so verify against Semrush guidance where authenticity matters.
- robots.txt token: SemrushBot
- Third-party SEO crawler (not a search engine)
- Documented to respect robots.txt
How it appears in analytics and logs
A request carrying the SemrushBot token is the Semrush SEO crawler fetching a URL — a bot event from a third-party tool, not a search engine and not a human visit. It does not influence search rankings by itself.
Diagnostic use case
Recognise SemrushBot as third-party SEO crawling and allow or exclude it via robots.txt using its token.
What WebmasterID can help detect
WebmasterID classifies SemrushBot server-side as an SEO crawler and shows its activity separately from human traffic and search-engine crawlers, so third-party SEO crawling is visible without log parsing.
Common mistakes
- Assuming SemrushBot crawling affects search rankings — it does not.
- Treating SEO-tool crawl hits as human visits.
- Blocking SemrushBot without realising it removes you from that tool's data.
Privacy and accuracy notes
Identification uses the user agent — no human identity. WebmasterID records SemrushBot as a bot event, separate from human analytics.
Related pages
- AhrefsBot — Ahrefs SEO crawler
AhrefsBot is the crawler operated by Ahrefs to build its SEO and backlink index. It is a third-party crawler, not a search engine, so it does not affect Google or Bing rankings directly. It uses the AhrefsBot robots.txt token and is documented as respecting robots.txt and crawl-delay.
- DotBot — Moz's web crawler
DotBot is the crawler operated by Moz to build its link index, which powers Moz's link-analysis tools. It is a third-party crawler, not a search engine. It uses the DotBot robots.txt token and is documented as honouring robots.txt.
- Bot vs human
Separate automated crawlers from real human visits.
Sources and verification notes
- Semrush — SemrushBot documentationDocuments SemrushBot variants and robots.txt handling.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.