METHOD DOSSIERMSC-P-004Evidence foundationsVerified scientific dossier

Statistical significance or effect size: which result should be interpreted?

A p-value measures compatibility with a null model. Effect size describes observed magnitude and should be paired with an interval and a practical relevance threshold.

Direct answer

Assess detection, magnitude and precision together.

A p-value measures compatibility with a null model. Effect size describes observed magnitude and should be paired with an interval and a practical relevance threshold.

American Statistical Association, 2016 Lakens, 2013

The decision rule

Always interpret four objects together: the effect estimate, its interval, the p-value from the declared test, and a practical-importance threshold fixed before seeing the result. A small p-value measures neither magnitude nor usefulness; a large p-value does not demonstrate absence.

American Statistical Association (2016)

Three reading levels

1
Decision-maker: ask whether the interval clears the business threshold, not only whether p is below 0.05.
2
Practitioner: report absolute difference, relative lift, interval, p-value, sample sizes and multiplicity rule.
3
Analyst: define the estimand, sampling process, δ, interval level and sensitivity analysis.

Concrete marketing situation

A team compares two conversion rates and must decide whether the difference justifies rollout. It has a p-value, but change costs require at least +1 conversion point. The real question is: which magnitudes remain compatible with the data, and do they clear that threshold?

Exact scientific question

For the declared populations, units and windows, what is the descriptive difference Δ = πB − πA, with what precision, and is its interval compatible with zero, a useful effect δ, or an equivalence zone [−δ, +δ]?

Why p < 0.05 is insufficient

With a very large n, a tiny difference can yield a small p-value without economic value.
A nonsignificant result may simply be too imprecise to distinguish absence, harm or useful benefit.
Choosing δ after seeing the data turns the business rule into an after-the-fact justification.

American Statistical Association (2016) Lakens, Scheel & Isager (2018)

Intuition before equations

The p-value answers a compatibility question under a declared null model; it does not give the probability that H0 is true.
Effect size gives magnitude on a chosen scale; the interval shows values still compatible with the data and model.
Threshold δ translates cost, margin or risk into a minimum useful difference; it is not a universal constant.

Lakens (2013)

Required data

Two independent binomial groups, sample sizes nA/nB and conversions yA/yB; no causal assignment is claimed.
Same conversion definition, same observation window and explicitly comparable populations.
Practical threshold δ = 1 point fixed before analysis; alpha = 5% and test family declared.

Formal model and symbols

Effect

πA,πB; Δ=πB−πAp̂A=yA/nA; p̂B=yB/nB; Δ̂=p̂B−p̂A; lift=Δ̂/p̂A

nA,nB: units; yA,yB: conversions; π: population parameter; p̂: observed rate; Δ: estimand; Δ̂: estimator; δ: minimum useful difference fixed before analysis.

Uncertainty

H0:Δ=0; p̂pool=(yA+yB)/(nA+nB); SE0=√[p̂pool(1−p̂pool)(1/nA+1/nB)]; z=Δ̂/SE0; p=2Φ(−|z|)SE=√[p̂A(1−p̂A)/nA+p̂B(1−p̂B)/nB]; CI95=Δ̂±z0.975SE; CI90=Δ̂±z0.95SETOST: H01:Δ≤−δ; H02:Δ≥+δ; max(p1,p2)<α

α is the declared type-I error rate; Φ is the standard-normal CDF; zq is its q quantile. The Wald CI is used here only for large counts; prefer a score method when the approximation is fragile.

Fagerland, Lydersen & Laake (2015) Lakens, Scheel & Isager (2018)

Declared calculation, step by step

01
Fix population, window, metric, alpha, multiplicity and δ before results.
02
Compute rates, Δ in points and relative lift without changing units mid-interpretation.
03
Compute p under H0:Δ=0 and the 95% CI for Δ; for equivalence, also test [−δ,+δ] with two one-sided tests.
04
Classify the result against zero and δ, then document the decision and what it does not prove.

Four results that do not mean the same thing

Synthetic illustration — teaching values, not observed

Four results that do not mean the same thing
Scenario	Counts A → B	Rates A → B	Δ̂	Relative lift	p	IC95	IC90	TOST p	Decision
`detectable_negligible`	500000/50000 → 500000/51000	10.000% → 10.200%	0.200 pp	2.000%	0.0009046	[0.0819; 0.3181]	[0.1009; 0.2991]	1.62e-40	`detectable-but-negligible`
`inconclusive`	1000/100 → 1000/120	10.000% → 12.000%	2.000 pp	20.000%	0.152918	[-0.7411; 4.7411]	[-0.3004; 4.3004]	0.762700	`inconclusive`
`practically_convincing`	10000/1000 → 10000/1250	10.000% → 12.500%	2.500 pp	25.000%	2.21e-8	[1.6248; 3.3752]	[1.7655; 3.2345]	0.999609	`practically-convincing-benefit`
`practically_equivalent`	50000/5000 → 50000/5025	10.000% → 10.050%	0.050 pp	0.500%	0.792373	[-0.3223; 0.4223]	[-0.2624; 0.3624]	2.85e-7	`practically-equivalent`

The first scenario rejects both Δ=0 and effects of at least 1 pp: it is statistically detectable and contained in the practical-negligibility zone. This joint result never follows from the point estimate alone.

CSV Python · MIT R · MIT SPSS · MIT SAS · MIT

Validity assumptions

Are units independent within and between groups? If not, does the variance account for the actual dependence?
Does each unit have a binary outcome, a stable probability within its group, and one observation in the window?
Are counts large enough for the normal approximation? If not, is an appropriate binomial or score method used?
Does δ come from a documented value function fixed before analysis rather than from the observed result?

Diagnostics and uncertainty

Check all four binomial counts, denominators and absence of duplicate units.

Report the 95% CI for Δ and 90% CI for TOST; excessive width means insufficient information.

Repeat the reading for defensible δ values and the preregistered multiplicity correction.

Interpreting the result

Detectable but negligible: zero is excluded and the 90% CI lies entirely within [−δ,+δ]; the point estimate alone is insufficient.
Inconclusive: the interval contains both zero and useful effects; collect more data or decide under explicit risk.
Practically convincing: the 95% CI clears δ in the useful direction under the declared assumptions.
Practically equivalent: the 90% CI lies within [−δ,+δ] and both one-sided tests pass.

Supported and forbidden conclusions

Supported

Describe magnitude and precision for observed populations.
State which useful or negligible values remain compatible with the data.

Forbidden

Turn significance into importance, probability of H0 or causality.
Interpret p ≥ 0.05 as evidence of absence without a justified equivalence test.

Possible marketing decision

01
Roll out only if the plausible range clears the value threshold and operational risks are acceptable.
02
Stop for futility only when practical equivalence is supported around a defensible δ.
03
Otherwise treat the result as inconclusive and choose among more data, a bounded pilot or a risk-based decision.

When to use, and when not to use

Use

Comparisons of two independent proportions already justified by the analysis plan.
Decision reading after checking design, data and multiplicity.

Do not use

To choose the test: see MSC-P-039.
For power and n planning: see MSC-P-012; for causality: see MSC-P-013.

Reproducible implementations

Python 3.13 · reference

python msc-p004-reference.py --csv msc-p004-significance-effect-size.csv

R 4.5 · reference

Rscript msc-p004-reference.R msc-p004-significance-effect-size.csv

SPSS 31 · secondary

INSERT FILE='msc-p004-reference.sps'.

SAS 9.4 · secondary

%include 'msc-p004-reference.sas';

Python and R are the verified executable references. The secondary SPSS and SAS syntaxes apply the same counts, formulas and decision rules; run them in their native environments before operational use.

Expected final deliverable

01
Question, population, unit, window and design.
02
Absolute Δ, relative lift, p-value, 95% CI and preregistered δ.
03
Decision class, assumptions, multiplicity, limits and impossible decision.

Scientific sources and evidence level

American Statistical Association (2016)Institutional statement
The p-value measures neither size nor importance and should not decide alone.
Lakens (2013)Methodological article, CC BY
Effect sizes communicate magnitude and support comparison, synthesis and planning.
Lakens, Scheel & Isager (2018)Methodological tutorial, open access
Equivalence requires bounds based on the smallest effect of interest and two one-sided tests.
Fagerland, Lydersen & Laake (2015)Methodological review
The Wald CI for two proportions requires large counts and remains less robust than score intervals.

Method connections

Parent territoryStatistical decision methods: choose, quantify, validate