Analysing the Search Console Crawl Stats report
The Crawl Stats report in Google Search Console (under Settings) shows how Googlebot crawled your site over the last 90 days: total crawl requests, total download size, average response time, and breakdowns by response code, file type, crawl purpose (discovery vs refresh), and Googlebot type. Reading it well tells you whether crawling is healthy and where it is being wasted.
What the report shows
The Crawl Stats report lives under Settings in Google Search Console and covers roughly the last 90 days of Googlebot activity. At the top it shows three trends: total crawl requests, total download size, and average response time. Rising response time or falling requests can indicate the server is struggling, which leads Google to slow its crawl.
Below that, the report breaks crawling down by response code (200s, 3xx, 4xx, 5xx), by file type (HTML, images, CSS, JavaScript, and others), by crawl purpose (discovery of new URLs versus refresh of known ones), and by Googlebot type (such as smartphone, desktop, or resource-fetching agents).
- Trends: total requests, total download size, average response time
- By response code, file type, purpose (discovery vs refresh), Googlebot type
- Host status section flags robots.txt, DNS, and server connectivity problems
How to read it for problems
Healthy crawling looks like steady or growing 200 responses with a stable, low average response time. Warning signs include a climbing share of 5xx or 429 responses (the server is failing or rate-limiting Googlebot), a rising average response time (the server is slow, so Google crawls less), or crawl heavily skewed toward non-content file types or low-value URLs.
The report's host status indicators are especially useful: they call out robots.txt fetch failures, DNS resolution problems, and server connectivity errors that can throttle or halt crawling entirely. A failed robots.txt fetch can cause Google to pause crawling until it succeeds.
Turning it into action
If response time is high, work on origin and CDN performance so Google can crawl more per unit of load. If 4xx is rising, find and fix the broken or removed URLs (returning 410 for truly gone pages). If 5xx or 429 is rising, address capacity or overly aggressive rate limits. If discovery crawling is low for a site that publishes frequently, check internal linking and sitemaps.
Crawl Stats is a summary, not a per-URL log. To see exactly which URLs were involved, pair it with a server log audit and the URL inspection tool for individual pages.
How it appears in analytics and logs
Crawl Stats summarises Googlebot's requests to your site. Spikes in response time, growth in 4xx/5xx responses, or crawl concentrated on low-value file types or URLs all point to issues worth investigating before they affect indexing.
Diagnostic use case
Use the Crawl Stats report to confirm Googlebot is crawling at a healthy rate, spot rising error responses or slow response times, and see where crawl budget goes.
What WebmasterID can help detect
WebmasterID's server-side crawler records complement Crawl Stats: where Google reports only Googlebot, WebmasterID shows other search and AI crawlers reaching the same pages, giving a fuller crawl picture.
Common mistakes
- Reading Crawl Stats as human traffic — it is Googlebot activity only.
- Ignoring a rising average response time, which causes Google to crawl less.
- Overlooking host status errors (robots.txt/DNS/connectivity) that throttle crawling.
- Expecting per-URL detail from a report that is a site-level summary.
Privacy and accuracy notes
Crawl Stats reports Googlebot activity, not human visitors. It contains no personal data. WebmasterID likewise records crawler activity without attaching it to any person.
Frequently asked questions
- Where is the Crawl Stats report?
- In Google Search Console under Settings, in the Crawling section. It covers about the last 90 days of Googlebot crawl activity for the property.
Related pages
- Crawl rate and server load
When crawlers request pages faster than your origin can comfortably serve, load rises. Compliant crawlers respond to 429 and 503 with Retry-After by slowing down, giving you a controlled way to protect the server. Google adjusts crawl rate automatically based on site responsiveness and offers a way to report rate problems.
- Auditing crawls with server log files
A server log file crawl audit reads raw access logs to see exactly how crawlers interact with your site: which URLs each bot fetched, what status codes they received, how often, and how much of your crawl is spent on low-value paths. Because logs record every request server-side, they reveal crawl behaviour that JavaScript analytics and sampled reports cannot — the ground truth of who fetched what.
- Reading the Page Indexing (Coverage) report
The Page Indexing report (formerly Index Coverage) in Google Search Console shows how many of your pages are indexed and groups the not-indexed pages by reason — such as crawled-not-indexed, discovered-not-indexed, duplicate without user-selected canonical, excluded by noindex, blocked by robots.txt, redirect, or soft 404. Each reason points to a distinct fix.
- Website observability
See crawler requests and response codes across all bots, not just Googlebot, server-side.
Sources and verification notes
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.