The Two-Proportion z-Test
An A/B test compares two conversion rates measured on samples. The question is whether the gap between them is larger than the random variation you would expect from splitting the same audience two ways. The two-proportion z-test answers exactly that.
Formulas
SE = √(p̂(1 − p̂)(1/nA + 1/nB))
z = (pB − pA) ÷ SE
p-value = 2 × (1 − Φ(|z|))
95% CI = (pB − pA) ± 1.96 × SE(unpooled)
The pooled proportion is used for the test statistic because the null hypothesis assumes both variants share a single true rate. The confidence interval uses the unpooled standard error instead, since it is estimating an actual difference rather than testing against zero. That is why an interval can occasionally sit marginally either side of the significance verdict.
How Tests Go Wrong
The arithmetic is the easy part. Most bad A/B decisions come from how the test was run, not from how the numbers were crunched.
Peeking and stopping early
Checking the result repeatedly and stopping the moment it crosses p < 0.05 can push the real false positive rate well above 5%. Fix the sample size in advance, or use a sequential testing method designed for continuous monitoring.
Testing too many variants at once
Run twenty comparisons at the 5% threshold and one will look significant by chance alone. If you are testing several variants, tighten the threshold — the Bonferroni correction divides 0.05 by the number of comparisons.
Ignoring the business significance
With a large enough sample, a 0.05 percentage point difference becomes statistically significant while being worth nothing. Decide in advance what size of lift would justify shipping the change, and check the confidence interval against that number, not against zero.
How Much Traffic You Need
Required sample size rises sharply as the effect you want to detect gets smaller — roughly with the inverse square. Detecting a 10% relative lift needs about four times the traffic of detecting a 20% lift, and a low baseline conversion rate increases the requirement further.
As a rough guide, a site converting at 5% typically needs on the order of tens of thousands of visitors per variant to reliably detect a 10% relative improvement. If your traffic cannot support that, test bigger changes rather than small refinements — the alternative is a series of underpowered tests whose results are mostly noise.