Detecting automation from user agents
You can use the user agent as a first signal for spotting automation — tool tokens, headless markers, missing strings — but it is never conclusive, because any client can change it. Reliable detection pairs the UA with verification and behaviour, and records honest unknowns. This page explains a sound approach.
What the user agent can and cannot tell you
The user agent is a useful first signal. A named tool token (curl, python-requests, Go-http-client), a headless marker, or a missing string all raise the probability that a request is automated. A self-identifying crawler token tells you a declared bot is claiming to visit.
What it cannot do is prove a human. A browser-shaped string can be reproduced by a scraper, so the absence of an automation marker is not evidence of a person. The UA is a hint, not a verdict.
- Tool, headless, or missing UA: raises automation likelihood
- Self-identifying crawler token: a declared bot's claim
- Browser-shaped string: never proof of a human
Pair the UA with verification and honest unknowns
Robust detection layers signals. For declared crawlers, confirm the claim with reverse DNS or published IP ranges rather than trusting the token. For other clients, weigh behaviour — request rate, paths, and rule compliance — alongside the user agent.
Crucially, when the signals do not add up to a confident answer, record an honest unknown rather than forcing a bot-or-human label. A guessed classification corrupts analytics; an honest 'other' bucket keeps them trustworthy.
How it appears in analytics and logs
A tool, headless, or missing user agent raises the likelihood of automation, and a self-identifying crawler token plus verification confirms a known bot. But a browser-shaped string proves nothing on its own, since automation can wear one.
Diagnostic use case
Use the user agent as one input among several when separating automation from people, without over-trusting it or forcing a label onto ambiguous traffic.
What WebmasterID can help detect
WebmasterID combines server-side user-agent classification with crawler verification and keeps unconfirmed clients in an honest 'other' bucket, so automation detection does not rest on the UA alone.
Common mistakes
- Treating a browser-shaped user agent as proof the client is human.
- Trusting a crawler token without verifying it against IP or reverse DNS.
- Forcing a bot-or-human label onto ambiguous traffic instead of marking it unknown.
Privacy and accuracy notes
Automation detection uses the user agent, network-level verification, and behaviour — not visitor identity. WebmasterID records the outcome as a bot-or-human signal and never builds a human profile from it.
Related pages
- Spoofed and fake user agents: what to watch for
Spoofing a user agent is trivial — any client can claim to be Googlebot or a normal browser. This page explains why spoofing happens, the common fake-crawler patterns, and the verification methods that turn a claimed identity into a confirmed one.
- Headless browser user agents
Headless browsers run a real browser engine without a visible window, and are widely used for testing and scraping. Headless Chrome historically exposed a HeadlessChrome token, though automation can also wear an ordinary browser string. This page explains the patterns and why headless traffic is automation.
- Bot vs human traffic
The bigger picture on separating automation from people.
Sources and verification notes
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.