How do you design an A/B test that actually estimates an effect?

How many units per arm must be randomized to detect a prespecified absolute difference in a binary outcome at chosen error rates?

The unit is the eligible randomization unit analysed in its assigned arm over a fixed window. User, household, store and geography are not interchangeable.

The primary estimand is the absolute ITT difference p₁−p₀ for a binary conversion. The 0.7-point MDE, or 8.75% relative, is a decision choice, not a predicted effect.

Declare eligible population, credible baseline rate, MDE, primary metric and window, alpha, power, allocation ratio, attrition, clustering, contamination and traffic constraints.

Assume two independent arms, 1:1 allocation, binary outcome, two-sided test, adequate normal approximation, ITT analysis and no unit reuse. Clustering and repeated measures need adapted calculations.

n≈[z₁₋α/2√(2p̄(1−p̄))+z₁₋β√(p₀(1−p₀)+p₁(1−p₁))]²/(p₁−p₀)², where p̄=(p₀+p₁)/2.

With p₀=0.08, p₁=0.087, α=0.05 and power 0.80: z=1.959964 and 0.841621. The script calculates 24,515.438 then rounds upward.

Raw size: 24,516 units per arm. After a 1.15 design effect and 10% attrition: 31,326 per arm, 62,652 total.

Sample size is sensitive to baseline rate and MDE. Recalculate over plausible ranges, round upward and treat traffic and attrition uncertainty as feasibility risk.

Freeze population, unit, variants, primary metric, window, MDE, size, stopping rule, exclusions, ITT analysis, missing-data handling and sample-ratio checks.

After launch check assignment integrity, sample-ratio mismatch, contamination, differential attrition, outcome availability, exposure and change log.

Common errors are choosing MDE to obtain a small sample, peeking, multiplying metrics, randomizing users but analysing sessions, stopping at p<0.05 or excluding after assignment.

Power of 80% means an 80% probability of rejecting H0 under the exact planning effect and assumptions, not an 80% chance the variant is better.

Decide whether the test is feasible, which MDE is decision-useful and how much traffic and time to reserve before launch.

Calculated size proves neither absence of bias, metric relevance nor a real effect. It cannot justify causal conclusions when randomization or ITT is compromised.

Deliver question, population, unit, variants, metric and window, absolute and relative MDE, alpha, power, formula, adjustments, traffic, duration, diagnostics, analysis and decision rule.