Crawl rate and server load
When crawlers request pages faster than your origin can comfortably serve, load rises. Compliant crawlers respond to 429 and 503 with Retry-After by slowing down, giving you a controlled way to protect the server. Google adjusts crawl rate automatically based on site responsiveness and offers a way to report rate problems.
When crawl rate strains the origin
Crawlers fetch many URLs, and a large or frequently-updated site can attract substantial crawl volume. If that volume outpaces what your origin can serve comfortably, response times rise and errors can appear. The goal is to let legitimate crawling proceed while protecting the server during peaks.
The primary levers are HTTP back-off signals the crawler understands, not blunt blocking that could de-index content.
Back-off signals and Search Console
Compliant search and AI crawlers treat 429 Too Many Requests and 503 Service Unavailable as instructions to slow down, especially when paired with a Retry-After header indicating when to return. This lets you shed load temporarily without telling crawlers the content is gone.
Google additionally adjusts Googlebot's crawl rate automatically based on how your site responds: consistent fast responses can allow more crawling, while slow responses or server errors lead Google to back off. Google's documentation describes how to report a crawl-rate problem if Googlebot is crawling too aggressively. Use these mechanisms rather than hard-blocking, which risks dropping pages from the index.
- 429/503 with Retry-After tell compliant crawlers to slow down
- Google auto-adjusts crawl rate to your site's responsiveness
- Search Console documents reporting crawl-rate problems
Operator checklist
Correlate load spikes with crawler traffic. Return 429 or 503 with Retry-After to shed load temporarily rather than blocking. Keep your origin responsive so automatic crawl-rate tuning works in your favour. For persistent Googlebot over-crawling, use Google's documented reporting path.
How it appears in analytics and logs
Rising load correlated with crawler traffic signals crawl rate exceeding comfortable capacity. Returning 429 or 503 with Retry-After tells compliant crawlers to slow down; persistent server errors can cause Google to reduce crawl rate on its own.
Diagnostic use case
Manage crawler load on your origin using 429/503 back-off signals, and understand how Google adapts crawl rate to server responsiveness.
What WebmasterID can help detect
WebmasterID can surface which crawlers are driving request volume and whether they receive 429/503 responses, helping you tell heavy-but-legitimate crawling from abusive automation.
Common mistakes
- Hard-blocking crawlers under load instead of returning 429/503 with Retry-After.
- Omitting Retry-After, so crawlers do not know when to return.
- Letting slow responses persist, which can reduce overall crawl coverage.
Privacy and accuracy notes
Crawl-rate diagnosis concerns request volume and status codes, not personal data. WebmasterID reports crawler load patterns without exposing individual visitors or raw IPs.
Related pages
- HTTP 429 Too Many Requests and crawl rate
429 Too Many Requests means the client has sent too many requests in a given time and is being rate limited. It can include a Retry-After header telling the client when to try again. Compliant crawlers slow down in response, making 429 a controlled way to manage crawl rate.
- HTTP 503 Service Unavailable for maintenance
503 Service Unavailable means the server is temporarily unable to handle the request, usually due to maintenance or overload. It is the correct, index-protecting status for planned downtime: with a Retry-After header, compliant crawlers understand the outage is temporary and come back later.
- Bot intelligence
See which crawlers and automation are driving request volume.
Sources and verification notes
- Google Search Central — Reduce the Googlebot crawl rateDocuments crawl-rate adjustment and reporting over-crawling.
- MDN — 429 Too Many Requests
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.