HTTP status code cheat sheet for crawlers
This cheat sheet maps the five HTTP status classes to what they mean for crawlers and indexing. It is a quick reference for reading server logs and Search Console crawl data: which codes index normally, which redirect, which signal client errors, and which are server failures crawlers will retry. The aim is to interpret status codes through a crawl-and-index lens rather than a generic one.
How the five classes map to crawling
HTTP groups status codes into five classes, and each has a distinct meaning for a crawler. 1xx are interim/informational and rarely affect indexing directly. 2xx are success — a 200 is the normal indexable response. 3xx are redirects that move a crawler (and link equity) toward a target. 4xx are client errors signalling the request itself was rejected or the resource is gone. 5xx are server errors the crawler treats as temporary, backing off and retrying later.
Reading a log or crawl report by class first, then by specific code, is the fastest way to triage what crawlers are encountering.
- 2xx success — 200 indexes; 204/206 are special-purpose successes
- 3xx redirect — 301/308 permanent (move equity); 302/307 temporary
- 4xx client error — 404/410 gone; 401/403 blocked; 429 rate-limited
- 5xx server error — 500/502/503/504 transient; crawlers back off and retry
Codes that most affect indexing
A handful of codes carry outsized weight. 200 is the baseline for an indexable page. 301 and 308 are permanent redirects that consolidate signals onto the target; 302 and 307 are temporary and keep the original URL in play. 404 and 410 tell crawlers a page is gone — 410 is a stronger, more permanent signal. 429 and 503 ask crawlers to slow down or come back later, and Google treats sustained 5xx/429 as a reason to reduce crawl rate.
The most damaging patterns are soft 404s (a 200 on a page that is really missing) and long redirect chains, both of which waste crawl budget and confuse indexing. Match the status you return to the truth about the resource.
Using it for triage
When a page is not indexed, check the status a crawler actually received, not what a browser sees. A page that returns 200 to you but 403 or 503 to a crawler will not index. A page you deleted should return 404 or 410, not a 200 soft-404 or a 302 to the homepage.
Pair this cheat sheet with the per-code entries in this category for the precise semantics, and with the URL inspection tool and coverage report to confirm how Google specifically classified each URL.
How it appears in analytics and logs
Status code classes tell you how crawlers will treat a URL: 2xx success is indexable, 3xx moves equity to a target, 4xx is a client/access problem, and 5xx is a server failure crawlers back off from. Reading them by class speeds up triage.
Diagnostic use case
Quickly interpret the HTTP status codes a crawler logs — knowing which index, which redirect, which to fix, and which to retry — without re-reading every RFC.
What WebmasterID can help detect
WebmasterID records the status code returned to each crawler fetch, so you can see the distribution of 2xx/3xx/4xx/5xx that crawlers actually received, page by page, without parsing raw logs.
Common mistakes
- Returning 200 (soft 404) for pages that are actually missing.
- Using 302 temporary redirects for moves that are permanent (should be 301/308).
- Assuming a browser-visible 200 is what crawlers receive — verify the crawler's status.
- Ignoring sustained 5xx/429, which lead crawlers to reduce crawl rate.
Privacy and accuracy notes
Status codes describe responses to requests, not people. WebmasterID records the status of crawler fetches without attaching them to any visitor profile.
Frequently asked questions
- Which redirect should I use for a permanent move?
- Use 301 or 308 for permanent moves so crawlers consolidate signals onto the target. Reserve 302 and 307 for genuinely temporary situations where the original URL should remain canonical.
- Do 5xx errors hurt indexing?
- Occasional 5xx errors are treated as transient and retried. Sustained 5xx or 429 responses lead crawlers to slow down, and pages that stay unavailable can eventually drop from the index.
Related pages
- HTTP 200 OK: what it means for crawlers
200 OK means the request succeeded and the server returned the resource. For crawlers it is the green light to process and potentially index a page. The subtle trap is the soft 404 — an error or empty page served with a 200 status, which wastes crawl budget and pollutes the index.
- Soft 404 diagnosis and fixes
A soft 404 is a page that is effectively missing or empty but returns a 200 status, so it looks successful to crawlers while offering no real content. Search engines try to detect them, but you should not rely on that. Soft 404s waste crawl budget and can clutter the index with low-value URLs.
- Redirect best practices for crawlers
Good redirects keep crawlers and link equity flowing to the right destination. The essentials: use 301 or 308 for permanent moves and 302 or 307 only for genuinely temporary ones, redirect each old URL directly to its final target in a single hop, map to the closest equivalent page rather than dumping everything on the homepage, and avoid loops. These choices preserve indexing signals and conserve crawl budget.
- Website observability
See the distribution of status codes crawlers received, page by page, server-side.
Sources and verification notes
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.