Difference-in-differences for measurement
Difference-in-differences (DiD) is a quasi-experimental method that estimates the causal effect of an intervention — like turning a campaign on in some regions — by comparing how a treated group changed against how an untreated control group changed over the same time. By differencing out both pre-existing gaps and shared time trends, DiD isolates the incremental effect. This page explains the method, its key assumption, and where it fits in measurement.
How DiD works
Take two groups — one that receives the intervention (treated) and one that does not (control) — and measure each before and after. Compute the change for each group, then subtract the control's change from the treated group's change. That second difference is the estimated effect.
The logic is that the control captures whatever would have happened anyway — seasonality, market trends — so removing its change leaves the part attributable to the intervention.
The parallel-trends assumption
DiD's validity rests on the parallel-trends assumption: absent the intervention, the treated and control groups would have moved in parallel. If they were already diverging for other reasons, the estimate is biased.
Analysts check this by examining pre-period trends and choosing controls that tracked the treated group closely beforehand. DiD is widely used for geo-experiments and policy evaluation precisely because it needs a comparison group rather than full randomization.
- Effect = treated change minus control change
- Differences out shared time trends and fixed gaps
- Relies on the parallel-trends assumption
How it appears in analytics and logs
A DiD estimate is the treated group's change minus the control group's change; a non-zero difference, under its assumptions, indicates an incremental effect attributable to the intervention.
Diagnostic use case
Estimate the incremental effect of a marketing change when a clean randomized experiment is not possible but a comparable untreated control exists.
What WebmasterID can help detect
WebmasterID's observed, aggregated conversion and traffic counts by segment can supply the treated and control series a difference-in-differences analysis needs.
Common mistakes
- Using a control whose pre-period trend differed from the treated group.
- Ignoring events that hit one group but not the other.
- Treating a DiD estimate as causal without checking parallel trends.
Privacy and accuracy notes
DiD works on aggregated group outcomes, not individual tracking, making it a privacy-friendly causal method. This is educational, not legal advice.
Related pages
- Synthetic control method
The synthetic control method estimates causal impact by constructing a 'synthetic' version of the treated unit — a weighted blend of comparison units that closely matches its pre-intervention behavior. The gap between the real treated outcome and its synthetic counterfactual after the intervention is the estimated effect. It is widely used in geo-experiments where a single market is treated. This page explains the construction and its assumptions.
- Geo experiments for measurement
A geo experiment divides geographic regions into a treatment group (which sees a media change) and a control group (which does not), then compares outcomes between them. Because assignment is at the region level rather than the user level, geo experiments measure incremental effect without needing cookies, device IDs, or per-person attribution — making them a privacy-resilient complement to touch-based models.
- Incrementality testing: what attribution cannot tell you
Incrementality testing measures the lift a channel actually causes by withholding it from a control group and comparing outcomes. It answers the question every attribution model dodges: would this conversion have happened anyway? It is causal where attribution is merely correlational, but it requires deliberate experiment design.
- Attribution analytics
Aggregated segment series for quasi-experimental analysis.
Sources and verification notes
- NIST/SEMATECH e-Handbook of Statistical MethodsReference for experimental and comparison-group statistical methods underpinning DiD.
- Google — GeoexperimentsResearch methodology (open source)Documents comparison-based geo-experiment measurement related to DiD logic.
Last reviewed 2026-06-24. Facts are checked against primary/official sources where available; uncertain specifics are marked “Data not yet verified” rather than guessed.