HTTP 400 Bad Request for crawlers
400 Bad Request means the server refused to process the request because it appeared malformed — bad syntax, invalid headers, or a request the server cannot interpret. Seeing 400s for crawlers usually points at malformed URLs, encoding issues, or a misbehaving edge layer rather than the crawler itself.
What 400 means
400 Bad Request is a client-error status: the server is saying it cannot or will not process the request because something about it is malformed — invalid syntax, a header it cannot parse, or a request framing problem. The server did not get far enough to act on the resource.
For a well-behaved crawler hitting a normal URL, a 400 is unexpected and signals a problem on the request path.
Common causes when crawlers see 400
Malformed URLs in your own internal links or sitemap (stray characters, broken encoding) can produce 400s when crawled. Other causes include overly long or malformed query strings, invalid percent-encoding, and edge proxies or WAFs that reject requests on strict header rules.
Because 400 is about the request, not the resource, the fix is usually upstream: correct the links you publish or relax an over-strict edge rule.
- Malformed or wrongly encoded URLs in links/sitemaps
- Invalid or overlong headers / query strings
- Edge proxy or WAF rejecting on strict rules
Operator checklist
Reproduce the failing request and inspect the exact URL and headers. Fix malformed internal links and sitemap entries. Check whether an edge layer or WAF is returning the 400, and review its rules. Confirm valid URLs return 200 once the request issue is corrected.
How it appears in analytics and logs
A 400 means the server judged the request itself invalid and did not process it. For crawlers this is unusual and worth investigating: it often traces to malformed URLs in your own links, character-encoding problems, or an edge/WAF layer rejecting requests.
Diagnostic use case
Diagnose why a crawler receives 400s on URLs that should be valid, and rule out malformed links, bad encoding, or an over-strict proxy.
What WebmasterID can help detect
WebmasterID can highlight URLs where crawlers receive 400s, helping you trace malformed links or an edge configuration that is rejecting otherwise valid requests.
Common mistakes
- Assuming a 400 is the crawler's fault rather than a malformed request you generated.
- Publishing wrongly encoded URLs in links or sitemaps.
- Overlooking an edge/WAF rule that rejects valid requests.
Privacy and accuracy notes
Status codes carry no personal data. WebmasterID reports 400 patterns for crawler traffic without exposing individual visitors.
Related pages
- HTTP 403 Forbidden and blocked crawlers
403 Forbidden means the server understood the request but refuses to authorize it, and authenticating will not help. For crawlers, a 403 often signals over-blocking — a WAF, bot-management rule, or IP filter rejecting legitimate crawlers and quietly removing pages from being indexed.
- HTTP 404 Not Found: what it means for crawlers
404 Not Found means the server has no resource at that URL. It is the correct, healthy response for genuinely missing pages — crawlers expect some 404s. Problems arise when important pages 404 by accident, when removed pages should signal 410, or when 'not found' pages wrongly return 200.
- Website observability
Spot URLs where crawlers receive client-error responses.
Sources and verification notes
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.