How to block AhrefsBot in robots.txt
AhrefsBot is the crawler Ahrefs uses to build its backlink and SEO index. This page gives the robots.txt rule to disallow it and notes that Ahrefs documents support for both robots.txt rules and the crawl-delay directive, so you can slow rather than fully block it.
What AhrefsBot is
AhrefsBot crawls the web to build the link and content index that powers the Ahrefs toolset. It is a third-party SEO crawler — it does not influence Google or Bing search results, and it is not an AI training crawler. Some operators block it to reduce crawl load from tools they do not use.
The rule and crawl-delay option
Ahrefs documents that AhrefsBot honours robots.txt. To disallow it site-wide:
User-agent: AhrefsBot Disallow: /
Ahrefs also documents support for the crawl-delay directive, so if you want to slow rather than block it you can specify a delay in its group:
User-agent: AhrefsBot Crawl-delay: 10
Use one approach or the other depending on whether you want AhrefsBot gone or merely gentler.
- Token: AhrefsBot
- Honours robots.txt per Ahrefs docs
- Supports crawl-delay to slow rather than block
How it appears in analytics and logs
A request with the AhrefsBot token is Ahrefs building its index. It is third-party SEO crawling, not a search engine and not a human visit; treat sustained activity as crawl load to manage, not audience.
Diagnostic use case
Disallow AhrefsBot, or slow it with crawl-delay, to reduce SEO-tool crawl load without affecting search or AI crawlers.
What WebmasterID can help detect
WebmasterID classifies AhrefsBot as an SEO crawler separate from human traffic, so you can confirm a block or crawl-delay change took effect for the compliant crawler.
Common mistakes
- Assuming blocking AhrefsBot affects Google or Bing — it does not.
- Setting crawl-delay expecting Googlebot to obey it too (Google ignores crawl-delay).
- Misspelling the token — it must be exactly AhrefsBot.
Privacy and accuracy notes
Managing AhrefsBot is a crawl-load and policy choice expressed in a public file. It involves no visitor data.
Related pages
- How to block SemrushBot in robots.txt
SemrushBot is the crawler Semrush uses to build its SEO datasets. Semrush documents several specialised sub-bots under related tokens, so this page covers the base disallow rule and explains why you may need to target multiple tokens to cover the activity you care about.
- The crawl-delay directive in robots.txt
Crawl-delay is a non-standard robots.txt directive that asks a crawler to wait between requests. Support is uneven: Google does not use it and points to Search Console instead, while Bing and Yandex have historically honoured it. This page explains the directive and the safer alternatives.
- Bot intelligence
See SEO-crawler activity separate from human traffic.
Sources and verification notes
- Ahrefs — About AhrefsBotDocuments robots.txt and crawl-delay support for AhrefsBot.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.