The crawl-delay directive in robots.txt
Crawl-delay is a non-standard robots.txt directive that asks a crawler to wait between requests. Support is uneven: Google does not use it and points to Search Console instead, while Bing and Yandex have historically honoured it. This page explains the directive and the safer alternatives.
What crawl-delay does
Crawl-delay appears inside a user-agent group and asks that crawler to wait a number of seconds between requests, for example:
User-agent: Bingbot Crawl-delay: 10
It is not part of the original Robots Exclusion Protocol and is not specified in RFC 9309, so each crawler decides whether and how to interpret it.
Who honours it
Support is inconsistent. Google documents that Googlebot does not support crawl-delay and that you should manage Google's crawl rate through Search Console instead. Bing and Yandex have historically honoured crawl-delay. Because behaviour varies and can change, treat crawl-delay as a hint for the crawlers that implement it, not a universal control.
For crawlers that ignore it, the alternatives are the crawler's own rate-control tools or, for abusive non-compliant clients, server-side rate limiting.
- Not in RFC 9309 — non-standard
- Googlebot ignores it; use Search Console for Google
- Bing and Yandex have historically honoured it
How it appears in analytics and logs
Crawl-delay is a request to space out requests. Seeing crawlers ignore it does not mean misconfiguration — many, including Googlebot, simply do not implement it.
Diagnostic use case
Slow down crawlers that honour crawl-delay to ease server load, while using the right channel for crawlers that ignore it.
What WebmasterID can help detect
WebmasterID shows crawl frequency per crawler, so you can see whether a crawl-delay setting actually changed the request rate for the crawlers that honour it.
Common mistakes
- Setting crawl-delay and assuming Googlebot will slow down — it ignores the directive.
- Relying on crawl-delay to stop abusive non-compliant clients.
- Setting an extreme delay that starves search crawlers of recrawls.
Privacy and accuracy notes
Crawl-delay is a load-management directive in a public file. It involves no visitor data.
Related pages
- How to block AhrefsBot in robots.txt
AhrefsBot is the crawler Ahrefs uses to build its backlink and SEO index. This page gives the robots.txt rule to disallow it and notes that Ahrefs documents support for both robots.txt rules and the crawl-delay directive, so you can slow rather than fully block it.
- robots.txt basics: what it does and what it cannot do
robots.txt is a plain-text file at your site root that tells compliant crawlers which paths they may request. This page covers the directives, how user-agent groups are matched, and the limits that trip people up: robots.txt is advisory, it does not hide pages from search, and it is not a security boundary.
- Website observability
Watch crawl rate per crawler before and after a change.
Sources and verification notes
- Google — How Google interprets robots.txtStates Googlebot does not support crawl-delay.
- RFC 9309 — Robots Exclusion ProtocolStandard does not define crawl-delay.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.