Sample size calculator

This is a two-sided test of two proportions, powered for a fixed sample:

n = (z₁₋α/₂ + z₁₋β)² × [p₁(1 − p₁) + p₂(1 − p₂)] ÷ (p₂ − p₁)²

where p₁ is the baseline rate, p₂ is the rate you are testing for, α is one minus the confidence level and β is one minus the power. The two z values are the standard normal quantiles for those probabilities, computed here rather than looked up in a table of three values, so a confidence level of 93 per cent produces the answer for 93 per cent.

The result is per variant, rounded up to a whole visitor, and the total assumes two equally sized groups.

Baseline conversion rate is the rate the control converts at today, from your own data, over a period long enough to be stable. Every other number on the page is derived from it, so a baseline taken from an unusual week produces a plan for a test that was never going to work.

Smallest lift worth detecting is relative, and it is a business decision rather than a statistical one. It is not what you hope for; it is the smallest improvement that would change what you do. A test sized for a lift larger than the change can plausibly produce is a test designed to find nothing.

Confidence level is the flip side of how often a test like this one reports a difference that is not really there. At 95 per cent, one test in twenty of two identical variants will produce a "winner".

Statistical power is the other side of that error: how often a real effect of exactly the size specified is detected. It defaults to 80 per cent, which means a genuine improvement is missed one time in five — a convention and not a law.

The arithmetic is exact. The assumptions underneath it are the part that decides whether the answer means anything.

Fixed sample, analysed once. The confidence level applies to a single test at the end. Checking daily and stopping when the result looks good is a different procedure with a much higher false positive rate — by some margin the most common way online tests produce results that do not replicate. If you want to look continuously, the analysis has to be one designed for it, not this one.

Equal, independent groups. Each visitor belongs to one variant and is counted once. On an ad platform this is where tests quietly fail: two ad sets shown to overlapping audiences are not two independent samples, the same person can land in both, and the platform's own delivery optimisation may send systematically different people to each. A platform's built-in split test exists precisely to hold this constraint, and a hand-built test of two ad sets against the same audience usually does not.

A stable rate. The baseline has to stay put for the length of the test. A sale, a stockout, a seasonal shift or a landing-page change mid-flight breaks the comparison, and the test has to restart rather than continue.

One test at a time. Running several comparisons and reporting the best one multiplies the false positive rate by roughly the number of comparisons, whatever the confidence level says.

At a 3 per cent baseline, a 20 per cent relative lift, 95 per cent confidence and 80 per cent power, each variant needs 13,911 visitors — nearly 28,000 in total, for a change of six tenths of a percentage point.

That figure is usually a surprise, and it is the most useful output of the page, because it converts a vague intention to test into a concrete traffic requirement that can be checked against reality before anything is built. If the site sees 1,800 visitors a day, this test takes over two weeks with the whole site in it. If it sees 200, the test takes four months and the answer arrives long after it mattered.

Knowing that in advance is the point. A test that cannot reach its sample is not a cautious test; it is a test that will be stopped early and read anyway, which is worse than not running it.

There are four honest responses and only one of them is statistical.

Test a bigger change. A radical variant produces a larger effect and needs a smaller sample. Testing button colours on thin traffic is how teams spend a quarter learning nothing.

Test higher up the funnel. Click-through rates and add-to-cart rates are far larger than purchase rates, so differences there resolve with much less traffic — accepting that a proxy metric is a proxy.

Accept more risk deliberately. Lower the confidence level or the power, know what you have accepted, and write it down. A decision made at 80 per cent confidence is not invalid; it is a decision made at 80 per cent confidence.

Decide without a test. Most changes are not worth a controlled experiment. Make the change on judgement, make it reversible, and watch the outcome. That is not rigour abandoned; it is rigour spent where it can actually buy an answer.