Effect size
Effect size is the magnitude of a difference — for conversion, the absolute lift (e.g. 3.0% to 3.3% is +0.3 points) or the relative lift (+10%). It is distinct from significance: a p-value says whether an effect is plausibly non-zero, effect size says whether it is big enough to matter. The smaller the effect you want to catch, the more traffic you need, so effect size anchors test planning.
Absolute vs relative
For a rate metric, the absolute effect is the difference in percentage points (3.0% → 3.3% is +0.3 points). The relative effect expresses it as a fraction of baseline (+10%). Both describe the same change but read very differently, so state which you mean. Relative lift looks larger and is easy to misread when the baseline is small.
- Absolute: difference in percentage points
- Relative: difference as a fraction of baseline
- Always say which one a headline number is
Why it drives sample size
Sample-size and power calculations take the minimum effect you want to detect as an input: halving the target effect roughly quadruples the traffic required. So effect size is not just a result — it is a design choice made before launch (the minimum detectable effect). Reporting an effect without its confidence interval hides how precisely it was measured.
Significance and effect size answer different questions; report both.
How it appears in analytics and logs
A 'significant' result with a tiny effect size may not justify the change; a large estimated effect from little data is likely overstated.
Diagnostic use case
State the minimum effect worth acting on, report observed lift as both absolute and relative, and read the confidence interval around it.
What WebmasterID can help detect
WebmasterID supplies first-party conversion counts so you can report observed effect size as both an absolute and a relative figure.
Common mistakes
- Quoting relative lift without the baseline, exaggerating the change.
- Reporting an effect size without a confidence interval.
- Acting on a statistically significant but practically trivial effect.
Privacy and accuracy notes
Effect size is computed from aggregate rates, not individuals. No personal data is required.
Related pages
- Minimum detectable effect (MDE)
The minimum detectable effect (MDE) is the smallest change in your metric that an experiment is set up to detect reliably. It is an input you choose, not an output: a smaller MDE demands more traffic. Setting the MDE to the smallest difference that would actually matter to the business keeps experiments honestly sized.
- Confidence intervals for conversion metrics
A confidence interval gives a range of plausible values for a metric rather than a single point. A 95% confidence interval is constructed so that, over many repeats, that procedure captures the true value 95% of the time. Reporting an interval communicates uncertainty honestly — a conversion rate of 4% with a wide interval is a very different claim than a narrow one.
- Statistical power
Power is the probability that a test correctly rejects the null when a true effect of a stated size exists: power = 1 − β. It rises with sample size, with the size of the effect you want to catch, and with a looser significance threshold; it falls with higher metric variance. Underpowered tests waste traffic by failing to detect real wins, so power is planned before launch.
- WebmasterID docs
How conversion events feed your own analysis.
Sources and verification notes
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.