User agent in CDN and WAF rules
Content delivery networks and web application firewalls let you write rules that match the user-agent header to route, cache, rate-limit, or block requests. It is a convenient lever, but the user agent is self-reported and spoofable, so UA-based rules carry real pitfalls: cache fragmentation, blocking legitimate clients, and being trivially evaded by attackers.
What this means
Edge platforms expose the user-agent header as a matchable field. Common uses include routing certain clients to a variant, varying cache by device class, rate-limiting a noisy client family, or blocking an obvious abusive token. Because the rule runs at the edge, it is fast and centralised.
The catch is that the user agent is a claim. A cooperative client sends an honest one; an adversary sends whatever passes your rule. So UA matching shapes well-behaved traffic but is not a security boundary.
Pitfalls to plan for
Cache fragmentation is the quiet one: if you vary the cache on the user agent, you multiply cache entries and lower hit rates, since user agents are highly varied. Prefer a normalised device class or a Client Hint over the raw string when caching.
False blocks are the loud one: an allowlist of 'real browser' tokens blocks legitimate but unusual clients (niche browsers, accessibility tools, valid bots), while a blocklist is evaded by simply changing the string. Combine UA rules with stronger signals — verified bot identity, behaviour, IP verification for declared crawlers — rather than trusting the header alone.
- Varying cache on the raw user agent fragments the cache
- UA allowlists block unusual-but-legitimate clients
- UA blocklists are evaded by changing the string
How it appears in analytics and logs
A CDN or WAF rule keyed on the user agent acts on a self-reported header. It can shape traffic effectively for cooperative clients, but it neither reliably stops determined bots nor cleanly identifies clients, because the value can be copied.
Diagnostic use case
Decide when matching the user agent at the CDN/WAF layer is appropriate, and avoid the cache, blocking, and evasion pitfalls of UA-based edge rules.
What WebmasterID can help detect
WebmasterID observes how user agents appear in traffic and classifies them server-side, complementing edge rules with bot-vs-human context rather than relying on spoofable UA matches alone.
Common mistakes
- Treating a UA-based WAF rule as a security boundary against determined bots.
- Caching with Vary on the raw user agent and tanking hit rates.
- Allowlisting a fixed set of browser tokens and blocking legitimate clients.
Privacy and accuracy notes
Edge rules should read the user agent as coarse context, not assemble high-entropy detail. Country-level edge estimates are coarse and not exact location; no raw IPs are exposed as a feature.
Related pages
- Pitfalls of UA-based bot blocking
Blocking traffic by matching user-agent substrings is a tempting but flawed bot defence. Hostile clients simply spoof a browser user agent to slip past, while legitimate browsers, accessibility tools, and beneficial bots get caught by over-broad rules. UA blocklists are a weak, high-collateral control compared with behaviour and verification.
- Risks of user-agent allowlisting
User-agent allowlisting — permitting only requests whose user agent matches an approved list — is the inverse of blocklisting and shares its core flaw: the user agent is a client-controlled claim. It blocks legitimate browsers, new versions, and assistive tools while being trivially bypassed by anyone who copies an allowed string.
- User agent in rate limiting
Some setups apply rate limits per user-agent string. Because the user agent is client-controlled, abusers rotate or randomise it to dodge limits, while many legitimate clients share identical user agents and can be throttled together unfairly. The user agent is a poor primary rate-limit key; it works better as a secondary signal.
- Bot intelligence
Server-side bot-vs-human context to complement spoofable edge rules.
Sources and verification notes
- MDN — Vary header (cache implications)Cache fragmentation from Vary; UA-based edge-rule pitfalls described from common practice.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.