The Search Console Crawl Stats report
The Crawl Stats report is a Google Search Console feature that summarises Googlebot's crawling of your site over the last 90 days — total crawl requests, total download size, average response time, and breakdowns by response code, file type, crawl purpose (discovery vs refresh), and Googlebot type. It is the primary first-party place to understand how Google crawls a property.
What this means
Crawl Stats (under Settings in Search Console) summarises how Googlebot crawled your site over roughly the last 90 days. It reports total crawl requests, total download size, and average page response time, then breaks the requests down by response code, fetched file type, crawl purpose, and Googlebot type.
The crawl purpose split — discovery (new URLs) versus refresh (re-crawling known URLs) — and the host-status section (robots.txt fetch, DNS, server connectivity) are the most useful for diagnosing crawl problems.
How to read it
Watch the trend lines: a sustained drop in total crawl requests alongside rising average response time often means Google reduced crawl rate because your host was slow or returning errors. A spike in 5xx or timeout responses in the breakdown points at host availability.
Use it together with server logs and a bot-aware analytics view: Crawl Stats is Google's own count, so it is authoritative for Googlebot but says nothing about other crawlers.
- Location: Search Console → Settings → Crawl stats
- Window: roughly the last 90 days
- Breakdowns: response code, file type, purpose, Googlebot type, host status
How it appears in analytics and logs
A shift in Crawl Stats — fewer total requests, rising response times, or a surge of error responses — points to crawl-budget or host-availability issues. The breakdown by purpose and Googlebot type tells you what Google was fetching and why.
Diagnostic use case
Diagnose crawl-rate changes, spot host-status problems, and see whether Google spends crawl on the right URLs, using Google's own first-party crawl data rather than only server logs.
What WebmasterID can help detect
WebmasterID complements Crawl Stats by recording crawler activity server-side across all bots — not just Googlebot — so you can cross-check Google's first-party numbers against what reached your origin.
Common mistakes
- Reading Crawl Stats as traffic or rankings data — it measures crawling, not visits or position.
- Ignoring the host-status section when crawl requests fall.
- Assuming Crawl Stats covers all bots; it is Googlebot-only.
Privacy and accuracy notes
Crawl Stats reports Googlebot activity, not human visitors. It contains no personal data; it is Google's view of its own crawler against your property.
Related pages
- Googlebot crawl frequency
Googlebot's crawl frequency is governed by two forces Google describes as crawl capacity limit and crawl demand. Capacity reflects how much your server can handle without slowing down; demand reflects how interesting and fresh Google judges your URLs to be. Google removed the manual crawl-rate setting, so the rate is mostly automatic and responds to your site's health and value.
- Crawl budget for large sites
Crawl budget is the practical limit on how many URLs Googlebot will crawl on your site in a given period, set by crawl capacity and crawl demand. Google says most sites do not need to worry about it, but very large sites (hundreds of thousands of URLs) or sites with many auto-generated URLs should manage it so Google spends crawling on valuable pages, not duplicates and dead ends.
- Managing third-party SEO crawler load
Third-party SEO crawlers such as AhrefsBot and SemrushBot can generate significant request volume without contributing to search visibility. You can manage their load by targeting their tokens in robots.txt, using crawl-delay where the crawler supports it, and blocking those that bring no value to you.
- Website observability
Server-side visibility into how crawlers reach your origin.
Sources and verification notes
- Google Search Central — Crawl Stats reportCrawl budget and Crawl Stats interpretation guidance.
- Google Search Central — Crawl Stats report help
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.