How do you measure campaign incrementality with a control group?

For the eligible population and prespecified window, what conversion difference does the assignment policy produce, with what precision, and is that range compatible with the declared value threshold?

Population: units eligible before assignment. Unit: the customer, cookie, store or geography actually randomized. Horizon: 14 days in the example. Each unit remains in its assigned arm for analysis.

The primary estimand is the average effect of assignment, ITT = E[Y(1)−Y(0)]. Randomization targets baseline exchangeability; consistency, positivity and no interference remain necessary for interpretation.

Retain unit identifier, eligibility, assignment, outcome, observation status, exposure, window and exclusions decided before the test. Aggregated counts alone cannot diagnose duplicates, attrition or contamination.

The assignment sequence must be unpredictable and applied at the declared unit; treatment and control must be well defined; follow-up must not depend differentially on arm; one unit must not affect another unit’s outcome without an interference model.

For a binary outcome, p̂1=y1/n1 and p̂0=y0/n0. The primary effect is Δ̂=p̂1−p̂0. Lift is Δ̂/p̂0; incremental test-arm conversions equal n1×Δ̂. Always declare the denominator.

Read the CSV, verify two unique arms and denominators, recompute rates from counts, calculate Δ̂, its standard error and interval, then derive lift and incremental volume. No seed is involved: the example contains realized counts.

Control: 1,600/20,000 = 8.00%. Treatment: 1,900/20,000 = 9.50%. ITT = +1.50 pp; secondary lift = 18.75%; 300 incremental conversions in the treatment arm.

The normal approximation gives SE=0.2826 pp and 95% CI [0.946, 2.054] pp. With small counts or extreme rates, prefer a score interval; with clustered assignment, estimate variance at the randomization level.

The treatment arm has 17,400 exposures for 20,000 assignments. ITT retains all 20,000 units and estimates the assignment-policy effect. Restricting to exposed units changes the estimand and introduces post-randomization selection.

The reference data have no missing outcomes. In production, report missingness by arm, explain its mechanism and run a bounded sensitivity analysis. Differential deletion after assignment can break randomization’s benefit.

Check expected sample ratio, unit duplicates, pretreatment balance, tracking stability, contamination, interference and window compliance. These diagnose the protocol; they do not replace estimation.

Typical errors: analyze exposed users as if randomized, exclude nonconverters post hoc, call an absolute difference lift, multiply ITT by the wrong population, stop at the first favorable result or ignore multiplicity.

Under the declared protocol, the range is compatible with a positive policy effect for this population and window. It does not describe the effect among exposed users only, persistence beyond 14 days or transport to another campaign.

Compare the ITT lower bound with a value threshold fixed before analysis, then choose rollout, another bounded pilot or stop while including cost, risk and capacity. The decision remains human and conditional.

Do not attribute the effect to exposed users only, to one channel in a multichannel campaign, to demand creation rather than temporal displacement, or to future performance outside the observed population and window.