Googlebot crawl frequency
Googlebot's crawl frequency is governed by two forces Google describes as crawl capacity limit and crawl demand. Capacity reflects how much your server can handle without slowing down; demand reflects how interesting and fresh Google judges your URLs to be. Google removed the manual crawl-rate setting, so the rate is mostly automatic and responds to your site's health and value.
What this means
Google describes crawl rate as a balance of crawl capacity limit and crawl demand. Capacity is how many simultaneous connections and how fast a crawl your server can sustain without degrading; demand is how much Google wants to crawl a URL based on popularity and how often it changes.
Google retired the manual crawl-rate limiter in Search Console, so for most sites the rate is automatic. You influence it indirectly: a fast, healthy server raises the capacity ceiling, and fresh, valuable content raises demand.
What to do about it
If crawling is too aggressive and harming your server, Google's guidance is to return 500/503/429 responses temporarily so Googlebot backs off, rather than relying on a removed rate slider. If you want more crawling, improve server speed and publish or update genuinely valuable content.
Do not expect to force a fixed crawl rate; treat crawl frequency as a health-and-value signal you shape over time, not a dial you set.
- Crawl capacity limit: what your server can sustain
- Crawl demand: how much Google wants those URLs
- Manual crawl-rate setting was removed; rate is mostly automatic
How it appears in analytics and logs
A drop in crawl frequency often means reduced crawl demand (stale or low-value URLs) or reduced capacity (server errors or slowness). A rise usually means fresh, valuable content or recovered host health — not a guarantee of more traffic.
Diagnostic use case
Understand why crawl rate rises or falls, and what you can and cannot influence, before assuming a crawl change signals a penalty or a problem.
What WebmasterID can help detect
WebmasterID records Googlebot hits server-side over time, so you can see crawl-frequency trends per URL and correlate dips with origin errors, complementing the first-party Crawl Stats report.
Common mistakes
- Believing a crawl-rate slider still exists in Search Console.
- Reading a crawl-frequency drop as a ranking penalty.
- Permanently serving error codes to slow Googlebot instead of using them only as a temporary brake.
Privacy and accuracy notes
Crawl-frequency analysis concerns Googlebot behaviour and server health, not human visitors. No personal data is involved.
Related pages
- The Search Console Crawl Stats report
The Crawl Stats report is a Google Search Console feature that summarises Googlebot's crawling of your site over the last 90 days — total crawl requests, total download size, average response time, and breakdowns by response code, file type, crawl purpose (discovery vs refresh), and Googlebot type. It is the primary first-party place to understand how Google crawls a property.
- Crawl budget for large sites
Crawl budget is the practical limit on how many URLs Googlebot will crawl on your site in a given period, set by crawl capacity and crawl demand. Google says most sites do not need to worry about it, but very large sites (hundreds of thousands of URLs) or sites with many auto-generated URLs should manage it so Google spends crawling on valuable pages, not duplicates and dead ends.
- Crawler traps and how to avoid them
A crawler trap (or spider trap) is a structure that produces an effectively unlimited number of low-value URLs, such as an infinite calendar, faceted-filter combinations, or session IDs in URLs. Traps waste crawl budget, can dilute indexing signals, and make logs noisy. They are recognised in Google's crawl-budget guidance and are fixable with URL hygiene.
- Website observability
Track crawler activity and origin health over time.
Sources and verification notes
- Google Search Central — Managing crawl budget for large sitesCrawl capacity limit and crawl demand explained.
- Google Search Central — Reduce the Googlebot crawl rate
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.