Significance calculator

p̂ = (c₁ + c₂) ÷ (n₁ + n₂)

se = √( p̂(1 − p̂) × (1/n₁ + 1/n₂) )

z = (p₂ − p₁) ÷ se

and the two-sided p-value is 2 × (1 − Φ(|z|)), where Φ is the standard normal cumulative distribution.

The rates are pooled for the standard error because the test asks what would happen if both variants shared one true rate; that is the hypothesis being tested, so it is the rate the noise is computed from.

The observed lift is relative — variant over control, minus one — because that is how a result gets quoted in a meeting.

In the worked example, 420 conversions from 14,000 visitors against 504 from 14,000: rates of 3 and 3.6 per cent, a 20 per cent relative lift, a z-score of 2.81 and a p-value of about 0.5 per cent.

The z-score is the difference measured in standard errors — how far apart the two rates are, in units of how far apart they would ordinarily wander. The p-value converts that into a probability: if the variants were identical, a gap this wide or wider would appear about one time in two hundred.

That is a strong result, and it is worth stating precisely what it is not. It is not the probability that the variant is better. It is not the probability that the result is correct. It is a statement about how often noise produces a difference like this one, and nothing else.

The arithmetic is exact and the assumptions are where tests fail.

One visitor, one variant, counted once. Duplicate counting, a person landing in both groups across devices, or a bot counted as traffic all break it.

Independent assignment. This is the assumption that most often fails on an ad platform. Two ad sets pointed at the same audience are not two independent samples: the same people are eligible for both, the audiences overlap, and the platform's own delivery optimisation may send systematically different people to each. A difference produced that way is real in the data and says nothing about the creative. The platform's own split-test tool exists to hold this constraint; two ad sets built by hand usually do not.

A stable rate. The underlying conversion rate has to hold for the length of the test. A sale, a stockout, a price change or a site release mid-flight means the two variants ran in different conditions.

A sample decided in advance, analysed once. The p-value is only what it says for a single analysis at a pre-declared sample size.

A p-value recomputed every day on a growing sample will cross 5 per cent by chance far more often than one time in twenty, because each check is a fresh opportunity for noise to wander over the line. Stopping at the first favourable moment therefore produces a great many "significant" results that do not replicate, and it is by some distance the most common methodological failure in online testing.

It is also the most tempting, because every testing dashboard shows a live significance figure and the figure moves every day.

There is a second version of the same mistake that is harder to see: running several variants, or several metrics, and reporting whichever crossed the line. Twenty comparisons at a 5 per cent threshold will produce one significant result on average from nothing at all, and the one that crossed is the one that gets written up. Decide the primary metric and the comparison before the test, and treat everything else the test produces as a question for next time rather than a finding.

The discipline is to size the test in advance, write the end date down, and read it once. If continuous monitoring is genuinely necessary — a variant can be actively losing money — then use a method designed for it and accept that it needs more data for the same confidence. What does not work is sizing a test one way and reading it another.

Significant and large. Ship it, and note the size of the effect as well as the p-value: a significant 2 per cent lift and a significant 30 per cent lift are the same verdict and very different decisions.

Significant and small. Ask whether the effect is worth the change, and whether the test was large enough that a small effect is what it was designed to find. Very large samples make trivial differences significant.

Not significant, large lift. Underpowered. The test has not yet distinguished the two, which is not the same as showing they are equal.

Not significant, small lift. The most common outcome and the least discussed. It is evidence that if there is an effect, it is probably smaller than the one the test was sized to find.

Negative. Check the p-value before concluding the variant is worse, for exactly the same reason a good early result is not a win.

One last habit: record the test with its sample, its dates, its p-value and what was decided. A result nobody wrote down gets re-litigated in six months by somebody who remembers it differently, and the re-run costs more than the record would have.