Delta method for ratio metrics
Many experiment metrics are ratios where the denominator is itself random — clicks per session, revenue per user, pages per visit. When the randomisation unit is coarser than the denominator unit, the numerator and denominator are correlated, so naive variance formulas are wrong. The delta method uses a first-order Taylor expansion to approximate the variance of the ratio correctly, fixing confidence intervals.
Why ratio variance is tricky
For a metric like clicks-per-session computed across users, both the total clicks (numerator) and the total sessions (denominator) vary from user to user and are correlated. Treating the ratio as if the denominator were fixed understates its variance. The delta method linearises the ratio around its mean with a Taylor expansion, yielding a variance that includes the variance of the numerator, the variance of the denominator, and their covariance.
- Numerator and denominator are both random and correlated
- Naive formulas understate variance → false significance
- Delta method = Taylor approximation of the ratio's variance
How it is used in practice
In experiment analysis the delta method is the standard fix for 'metric of analysis differs from unit of randomisation' problems — it gives an approximate but well-behaved variance for ratio metrics so significance and confidence intervals are trustworthy. It pairs naturally with picking the right randomisation unit and is documented in large-scale experimentation references. For sums (not ratios) at the unit of randomisation, ordinary variance is fine and the delta method is unnecessary.
Use it for ratios; don't over-apply it where the simple estimator already holds.
How it appears in analytics and logs
Without it, ratio-metric confidence intervals are too narrow, producing false significance; the delta method widens them to the right size.
Diagnostic use case
Apply the delta method when a metric is a ratio of two random quantities and the unit of analysis differs from the denominator's unit.
What WebmasterID can help detect
WebmasterID's first-party aggregates supply the sums and covariance the delta method needs for correct ratio-metric intervals.
Common mistakes
- Using simple variance for a per-session ratio analysed across users.
- Applying the delta method where the metric is already a unit-level sum.
- Ignoring the covariance term between numerator and denominator.
Privacy and accuracy notes
The delta method works on aggregate sums and their covariance, not individual records.
Related pages
- Randomization unit
The randomization unit is the thing you randomly assign to control or treatment: a user, a session, a device, a cookie, or a cluster. The choice must match how you analyse and how users experience the change. Mismatches cause two classic failures — a user flipping variants between sessions (inconsistent experience) and analysing at a finer grain than you assigned (understated variance, false significance).
- Confidence intervals for conversion metrics
A confidence interval gives a range of plausible values for a metric rather than a single point. A 95% confidence interval is constructed so that, over many repeats, that procedure captures the true value 95% of the time. Reporting an interval communicates uncertainty honestly — a conversion rate of 4% with a wide interval is a very different claim than a narrow one.
- CUPED variance reduction
CUPED (Controlled-experiment Using Pre-Experiment Data) reduces the variance of an experiment metric by adjusting it with a covariate measured before the test — typically each user's own pre-period behaviour. Because the covariate is independent of the treatment, the adjustment removes noise without introducing bias, so confidence intervals narrow and tests reach a decision with less traffic.
- WebmasterID docs
How conversion events feed your own analysis.
Sources and verification notes
- Deng, Knoblich, Lu — Applying the Delta Method in Metric Analytics (KDD), MicrosoftPeer-reviewed application of the delta method to experiment metrics.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.