HTTP 429 Too Many Requests and crawl rate
429 Too Many Requests means the client has sent too many requests in a given time and is being rate limited. It can include a Retry-After header telling the client when to try again. Compliant crawlers slow down in response, making 429 a controlled way to manage crawl rate.
What 429 means
429 Too Many Requests indicates the user has sent too many requests in a given amount of time (rate limiting). The response may include a Retry-After header indicating how long to wait before making a new request.
For crawlers, 429 is a polite 'slow down' rather than a permanent rejection.
429 and crawl rate
Compliant search and AI crawlers interpret 429 (and 503) as a signal to reduce their request rate. Returning 429 with a sensible Retry-After lets you protect your origin during load spikes without de-indexing content.
The risk is over-throttling: if legitimate crawlers are constantly rate limited, they may crawl less of your site, slowing how quickly new and updated pages are discovered. Set limits so genuine crawlers can still make progress.
- Retry-After tells compliant clients when to retry
- Crawlers back off rather than dropping the URL
- Persistent 429s can reduce crawl coverage over time
Operator checklist
Return 429 with a Retry-After when you must throttle. Make sure legitimate crawlers are not being constantly rate limited. Distinguish abusive automation (where 429 is appropriate) from verified crawlers you want indexing your site.
How it appears in analytics and logs
A 429 means a client exceeded a rate limit. For crawlers it is a back-off signal; compliant crawlers slow down. Persistent 429s for legitimate crawlers can reduce how much of your site gets crawled, so tune limits carefully.
Diagnostic use case
Throttle aggressive crawling without dropping content from the index, and confirm crawlers back off when you return 429 with Retry-After.
What WebmasterID can help detect
WebmasterID can surface which crawlers are receiving 429s, helping you tell whether rate limits are throttling legitimate crawlers or correctly slowing abusive automation.
Common mistakes
- Rate limiting legitimate crawlers so hard that crawl coverage drops.
- Returning 429 without a Retry-After header.
- Treating a temporary 429 spike as a permanent error.
Privacy and accuracy notes
Status codes carry no personal data. WebmasterID reports 429 patterns for crawler traffic without exposing individual visitors or raw IP addresses.
Related pages
- HTTP 503 Service Unavailable for maintenance
503 Service Unavailable means the server is temporarily unable to handle the request, usually due to maintenance or overload. It is the correct, index-protecting status for planned downtime: with a Retry-After header, compliant crawlers understand the outage is temporary and come back later.
- Crawl budget waste: causes and fixes
Crawl budget is the finite attention a search engine spends on your site. It is wasted when crawlers spend it on low-value URLs — endless faceted combinations, parameter variants, soft 404s, and redirect chains — instead of your important pages. Reducing that waste helps key content get crawled.
- Bot intelligence
See which crawlers and automation are hitting your origin.
Sources and verification notes
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.