User agents and A/B testing
A/B testing assigns visitors to variants to compare outcomes. The user agent matters twice: bot and crawler traffic should be excluded so automation does not pollute results, and assigning variants by user agent rather than at random biases the experiment. Clean UA filtering in, random assignment within — that is the safe pattern.
Keep bots out of experiments
Crawlers, monitors, and scripts hit your pages constantly, and if they land in an experiment they add conversions-or-not noise that no human produced. Because many of these clients self-identify in the user agent, the UA is a practical first filter for excluding them from the test population.
Filtering bot user agents before assignment keeps each arm representative of real humans. Without it, a crawl wave can swing a variant's numbers in ways that have nothing to do with the change you are testing.
Do not assign variants by user agent
It is tempting to bucket variants by browser or device using the user agent, but that breaks randomisation: the groups then differ by browser population, not just by the change under test, so any difference is confounded. Variant assignment should be random within the eligible population.
The user agent's proper role here is eligibility filtering (exclude bots, optionally scope to a platform), not assignment. If you genuinely want a per-platform experience, run separate experiments per platform rather than letting the UA decide the variant.
- Exclude crawler/monitor/script user agents before assignment
- Assign variants randomly, not by user agent
- Use the UA for eligibility filtering, not variant selection
How it appears in analytics and logs
If experiment data includes crawler user agents, automated hits inflate one arm without representing human behaviour. If variants are split by user agent, the groups differ systematically before the test even begins.
Diagnostic use case
Exclude bot traffic from experiments using the user agent, and avoid assigning variants by user agent in a way that biases results.
What WebmasterID can help detect
WebmasterID separates bot and crawler traffic from human visits server-side using the user agent, so you can keep automated hits out of experiment populations without hand-maintaining UA filter lists.
Common mistakes
- Leaving bot user agents in the experiment population.
- Splitting variants by user agent and confounding the result.
- Relying on a stale UA exclusion list instead of robust bot filtering.
Privacy and accuracy notes
Using the user agent to filter bots from an experiment is coarse client metadata, not personal data. Variant assignment should not rely on identifying individuals; the UA is a device/browser hint only.
Related pages
- User agent in analytics
Analytics platforms parse the user-agent string to report browser, operating system, and device-type breakdowns. Because the user agent is client-supplied, increasingly reduced, and easily spoofed — and because bots send their own strings — these breakdowns are useful approximations, not exact device censuses.
- Bot vs browser user agents: how to tell them apart
A user-agent string is a self-reported label, not an identity. This page explains how declared bots name themselves, why almost every UA still starts with the legacy Mozilla token, and how to read the difference between an automated client and a real browser without over-trusting the string.
- Detecting automation from user agents
You can use the user agent as a first signal for spotting automation — tool tokens, headless markers, missing strings — but it is never conclusive, because any client can change it. Reliable detection pairs the UA with verification and behaviour, and records honest unknowns. This page explains a sound approach.
- Bot vs human
Keep automated traffic out of experiment populations server-side.
Sources and verification notes
- MDN — User-Agent headerDefines the user-agent field used to filter bot traffic from experiments.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.