Variant B is up 18% after two days. The temptation is to call it, ship it, and move on. Two weeks later the change has produced nothing measurable, and nobody can explain why the test was so clearly right and the result so clearly absent.

What went wrong was not the tool or the tracking. It was stopping on the first favourable reading of a number that swings wildly while samples are small.

What does statistical significance actually mean?

It means the difference you measured would be unlikely to appear by chance if the two variants performed identically. It does not mean the difference is large, or that shipping it is worth the engineering time. Those are separate questions, and significance answers neither.

The threshold most teams use is a p-value below 0.05, which corresponds to 95% confidence. A p-value of 0.03 says that if the variants were truly the same, only 3% of repeated experiments would show a gap at least this big.

What a p-value is not

A p-value of 0.03 does not mean there is a 97% chance that B beats A. That reading is intuitive, widely repeated, and wrong. The p-value is calculated by assuming the variants are identical and asking how surprising your data would be under that assumption. It says nothing directly about the probability that your variant is better.

The distinction matters in practice because the intuitive reading invites you to treat a marginal result as near-certain, when a second run of the same test may well land on the other side of the line.

The arithmetic behind the verdict

Comparing two conversion rates uses a two-proportion z-test. Pool the two variants to estimate a shared rate, calculate how much the observed difference could be expected to wobble, and express the gap in units of that wobble.

  • Pooled rate p̂ = (conversions A + conversions B) ÷ (visitors A + visitors B)
  • Standard error = √( p̂ (1 − p̂) (1/nA + 1/nB) )
  • z = (rate B − rate A) ÷ standard error
  • Significant at 95% confidence when the absolute value of z passes 1.96

Take 5,000 visitors per variant, with 250 conversions on A and 300 on B — 5.0% against 6.0%. The pooled rate is 5.5%, the standard error is 0.00456, and z comes to 2.19. That gives a two-tailed p-value of 0.028, comfortably under the threshold. The A/B test significance calculator runs this on your own numbers and returns the confidence interval alongside the verdict.

How much traffic do you need?

Far more than most people expect, and the requirement grows sharply as the effect you want to detect gets smaller. Sample size scales with the inverse square of the difference, so halving the effect you are chasing roughly quadruples the traffic needed.

For a site converting at 5% that wants to detect a 10% relative improvement — 5.0% moving to 5.5% — at 95% confidence and 80% power, the requirement lands near 30,000 visitors per variant. At 20% relative improvement it drops to roughly 8,000. Nothing about the tooling changes that; it is a property of the arithmetic.

If your traffic cannot support those numbers, test bigger changes. A series of underpowered tests on button colours produces a series of results indistinguishable from noise, and the time spent running them is the real cost.

Why peeking breaks the test

Checking a test repeatedly and stopping the moment it crosses 0.05 does not give you a 5% false positive rate. Every look is another opportunity for random fluctuation to cross the line, and with frequent monitoring the true error rate climbs several times above the threshold you think you are holding.

Two defences work. Fix the sample size before the test starts and do not stop until you reach it, whatever the dashboard says on day three. Or use a sequential testing method designed for continuous monitoring, which adjusts the threshold to account for repeated looks. What does not work is the informal compromise of peeking often but promising to be strict about it.

Run for whole business cycles

Traffic on a Tuesday behaves differently from traffic on a Sunday, and paid traffic behaves differently from organic. A test that runs from Monday to Wednesday samples a biased slice of your audience even if it collects enough visitors.

Run for at least one full week, and preferably two, so every day of the week appears in both variants. If you run seasonal promotions or send a weekly newsletter, make sure the test window covers a representative pattern rather than straddling a spike.

Significance is not the same as worth shipping

With a large enough sample, a 0.05 percentage point difference becomes statistically significant while being worth nothing commercially. Significance testing answers whether an effect is real, not whether it matters.

Decide before the test what size of improvement would justify shipping the change, then check the confidence interval against that number rather than against zero. An interval spanning 0.2% to 4.1% is significant, but if you needed 2% to break even on the work, the test has not told you what you needed to know.

Frequently asked questions

My test is not significant. Does that mean the variants perform the same?

No. It means you do not have enough evidence to say they differ. A real and useful improvement can easily fail to reach significance on a small sample. Look at the width of the confidence interval — a wide one spanning zero means the test was underpowered, not that the change was worthless.

Should both variants get equal traffic?

An even split extracts the most statistical power from a given total, so it is the sensible default. Unequal splits are handled correctly by the maths. What matters far more is that assignment is random and that both variants run over exactly the same period.

Can I test more than two variants at once?

Yes, but tighten the threshold. Run twenty comparisons at 5% and one will look significant by chance alone. The Bonferroni correction divides 0.05 by the number of comparisons, so four variants tested against a control use 0.0125 as the threshold.

What is the difference between confidence and power?

Confidence controls how often you declare a winner that does not exist — the 95% figure. Power controls how often you detect a real effect when one is present, conventionally set at 80%. Most teams set confidence carefully and ignore power, which is why so many tests end inconclusive.

Why is the two-tailed test the default?

Because you rarely know in advance that a change can only help. A one-tailed test halves the p-value and is only defensible when a result in the opposite direction would lead to exactly the same decision as no difference at all, which is seldom true.

A significant lift in conversion rate still has to survive the economics: conversion rate means little without knowing your CAC, and industry benchmarks put any rate in context. For the campaign arithmetic that follows a winning test, the marketing calculators cover ROI, ad spend, and conversion economics.