Experimental Statistics: how to tell a real effect from noise
The question it answers. A landing page redesign goes live and conversion moves from 5% to 6%. Is the redesign working, are these visitors different, or is it random fluctuation? Users differ from each other, so even with no change, today's rate never exactly equals tomorrow's. Experimental statistics asks: how big a difference stops looking like fluctuation?
Step 1: random assignment. Split visitors randomly into A (old page) and B (new page). Randomization makes the two groups statistically identical in distribution: age, source, and device differences average out, and the only systematic difference left is the redesign. Without randomization, the split itself biases everything that follows. Limits: randomization guarantees comparability on average, not in any single run, and the gap is worst with small samples. It does not fix representativeness: the people in the experiment may differ from future users. And it breaks when users interfere with each other.
Step 2: the null hypothesis and the p-value. Start from the null hypothesis: the redesign does nothing, and the two true rates are equal. Under that hypothesis, the observed difference is pure sampling fluctuation. The p-value is the probability, under the null hypothesis, of seeing a difference this large or larger. Read it in the right direction: p = 0.03 is not "a 97% probability the redesign works", and not "a 3% probability the null hypothesis is true". The p-value is computed assuming no effect, so it cannot directly give the probability that the hypothesis is true.
Step 3: two errors and power. A Type I error is a false positive: declaring an effect when there is none. Its probability is the significance level, written α, conventionally 5%. That 5% is Fisher's convention, not a law of nature. A Type II error is a false negative: missing a real effect. Its probability is β. Statistical power is 1 - β, conventionally 80%: if the effect is real, the experiment catches it 80% of the time.
Step 4: sample size. Let n be the sample size per group, σ the standard deviation near the baseline, and δ the difference to detect, the Minimum Detectable Effect (MDE). A two-sided α = 0.05 gives z = 1.96, and 80% power gives z = 0.84. For two groups, n = 2σ²(1.96 + 0.84)² / δ², and (1.96 + 0.84)² is about 7.84, times 2 about 15.68, rounded to 16:
Worked example, with illustrative numbers, not real traffic. Baseline conversion 5%, so σ² = 0.05 × 0.95 = 0.0475. To detect a 1 percentage point lift, δ = 0.01, so n ≈ 16 × 0.0475 / 0.0001 = 7600 per group, about 15200 visitors in total. The denominator is δ², so halving the effect to detect quadruples the sample size. Set the MDE before running: the smaller the effect you want to detect, the more traffic you need. With too little traffic, either accept a bigger MDE or accept a longer run.
Peeking. Peeking means checking results repeatedly before the planned end and stopping at the first significant look. Each look is another lottery ticket: a nominal 5% false-positive rate inflates to about 14% at 5 looks and about 19% at 10 looks. The 5% promise only holds with a pre-committed single look, or with a formal sequential method that budgets α across looks. Checking a landing page experiment daily and stopping when it turns green is handing yourself extra tickets.
Novelty effect. Users' early reaction to something new does not represent the long run. A redesign can draw extra clicks from curiosity in the first days, or put users off because they dislike change; both directions mislead if early numbers are read as long-run effects. The fix is to run long enough and read the stable period after novelty wears off, not the opening days.
The synthesis: what survives, what is downgraded, what is dead. Kept: random assignment, still the only clean route to comparability, within its bounds; the square law, halving δ quadruples n, which is structural rather than conventional; and pre-commitment discipline, MDE, horizon, and primary metric fixed in advance, the only defence against peeking and multi-metric inflation. Downgraded: p-values, 5%, 80%, and the formula's exact n, from verdicts to references. The formula's n is an order-of-magnitude guide under ideal conditions, independent identical distribution, known variance, stable traffic. Real metrics are often skewed, for example revenue metrics dominated by a few large users, so variance runs larger than the formula assumes, and testing many metrics or many experiments at once inflates false positives further. Dead: deciding on a p-value crossing a line alone. The working rule: the framework is a design discipline plus a reference ruler, not a verdict. Decide on effect size, the confidence interval, the range the effect plausibly lies in, and the cost of being wrong. Not significant means not measured, never means no effect.