Incrementality testing: what attribution cannot tell you
Incrementality testing measures the lift a channel actually causes by withholding it from a control group and comparing outcomes. It answers the question every attribution model dodges: would this conversion have happened anyway? It is causal where attribution is merely correlational, but it requires deliberate experiment design.
What this means
An incrementality test deliberately withholds a channel (or audience) from a randomised control group, runs for a set period, and compares conversions between exposed and held-out groups. The difference estimates the channel's causal lift — the conversions it actually created.
Why it outranks credit splits
Every attribution model distributes credit among touches that were present; none asks whether the conversion would have occurred without them. Incrementality answers exactly that. The trade-off is cost and rigour: it needs proper randomisation, enough volume to detect an effect, and patience, and it estimates lift with confidence intervals rather than exact counts.
Use attribution for everyday navigation and incrementality to settle the high-stakes 'is this channel actually working?' questions.
- Compares exposed vs randomly held-out groups
- Estimates causal lift, not credit
- Needs randomisation, volume, and time
How it appears in analytics and logs
Incrementality results estimate causal lift with uncertainty. A channel with high attributed credit but low measured lift is largely taking credit for conversions that would have happened anyway.
Diagnostic use case
Run incrementality tests when a channel's attributed credit is high but you suspect it harvests demand that would convert regardless — only an experiment can tell.
What WebmasterID can help detect
WebmasterID's first-party, aggregate signals support before/after and holdout comparisons without user-level identity graphs.
Common mistakes
- Treating attributed credit as if it were incremental lift.
- Running a holdout without proper randomisation.
- Reading lift point estimates without their uncertainty.
Privacy and accuracy notes
Holdout experiments compare aggregate group outcomes and need no user-level cross-site tracking. This is educational, not statistical or legal advice.
Related pages
- Marketing mix modeling (MMM): top-down measurement
Marketing mix modeling (MMM) estimates how much each channel contributed to outcomes using aggregate, time-series data — spend, sales, seasonality — rather than user-level paths. It predates digital tracking, needs no cookies, and is gaining renewed interest as privacy limits user-level attribution. It is statistical inference, with real uncertainty.
- Data-driven attribution: promise and caveats
Data-driven attribution (DDA) assigns credit using a model trained on a site's own conversion paths rather than a fixed rule like last-click. Done well it credits assist touches more fairly. Its caveats are real: it needs enough conversion volume, it is a model not a measurement, and it cannot see touches that were never tracked.
- Conversion rate: definition and denominators
Conversion rate is the share of some base that converted. The trap is the denominator: conversions per session, per user, and per unique visitor give different numbers and mean different things. Without stating the base, a conversion rate is ambiguous — and comparing rates with different bases is meaningless.
- Attribution analytics
Pair credit views with holdout thinking.
Sources and verification notes
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.