SerpstatBot — Serpstat's web crawler
SerpstatBot is the crawler operated by Serpstat to collect backlink and SEO data for its platform. It is a third-party SEO tool crawler, not a search engine. Serpstat documents SerpstatBot and publishes robots.txt and crawl-rate guidance for operators who want to identify or restrict it.
What this means
SerpstatBot crawls the public web to build the backlink and SEO datasets behind the Serpstat platform. It is a third-party SEO tool crawler, not a search-engine indexer, so its visits do not affect search rankings.
You may see SerpstatBot whether or not you use Serpstat, because the tool analyses links and pages across the web. That is expected behaviour for SEO data crawlers.
How SerpstatBot identifies itself
SerpstatBot uses the robots.txt user-agent token SerpstatBot. Its user-agent string contains that token together with a self-identifying URL pointing at Serpstat's crawler documentation. Match on the stable token rather than a full version string.
The user agent is a claim that can be copied, so for requests where authenticity matters, corroborate against Serpstat's published documentation rather than trusting the string alone.
- robots.txt token: SerpstatBot
- User agent contains the SerpstatBot token plus a Serpstat URL
- An SEO/backlink-data crawler, not a search-engine indexer
robots.txt and crawl rate
SerpstatBot honours robots.txt. To disallow it site-wide:
User-agent: SerpstatBot Disallow: /
Serpstat documents crawl-delay support so you can throttle SerpstatBot rather than block it if load is the only concern. robots.txt is a request honoured by compliant crawlers, not an access-control mechanism.
How it appears in analytics and logs
A request carrying the SerpstatBot token is Serpstat's crawler fetching a URL to build backlink and SEO datasets — a bot event, not a human visit. It reflects competitive SEO data collection and should be counted as crawl coverage, not audience.
Diagnostic use case
Identify SerpstatBot in logs, confirm it is collecting backlink/SEO data, and slow or block it via robots.txt if its crawl load is unwanted.
What WebmasterID can help detect
WebmasterID classifies SerpstatBot server-side as an SEO crawler and shows its activity on the bot-intelligence surface, separate from human analytics, so you can see Serpstat data-collection hits without log parsing.
Common mistakes
- Treating SerpstatBot data-collection hits as human traffic.
- Assuming SerpstatBot indexes pages for a search engine.
- Blocking it entirely when crawl-delay would have addressed the load.
Privacy and accuracy notes
SerpstatBot detection uses only the request user-agent. No human identity is involved. WebmasterID records the crawl as a bot event, separate from human analytics, and never attaches it to a visitor profile.
Related pages
- SemrushBot — Semrush SEO crawler
SemrushBot is the crawler operated by Semrush to gather data for its SEO toolset. It is a third-party crawler, not a search engine, so it does not affect search rankings directly. It uses the SemrushBot robots.txt token and is documented as respecting robots.txt.
- AhrefsBot — Ahrefs SEO crawler
AhrefsBot is the crawler operated by Ahrefs to build its SEO and backlink index. It is a third-party crawler, not a search engine, so it does not affect Google or Bing rankings directly. It uses the AhrefsBot robots.txt token and is documented as respecting robots.txt and crawl-delay.
- Managing third-party SEO crawler load
Third-party SEO crawlers such as AhrefsBot and SemrushBot can generate significant request volume without contributing to search visibility. You can manage their load by targeting their tokens in robots.txt, using crawl-delay where the crawler supports it, and blocking those that bring no value to you.
- Web crawlers overview
How WebmasterID separates SEO data crawlers from search bots and humans.
Sources and verification notes
- Serpstat — SerpstatBot crawler documentationDocuments the SerpstatBot token, robots.txt, and crawl-rate handling.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.