Test duration calculator

Visitors needed per variant comes from the sample size calculator, or from whatever sizing method you used. It is per variant, not in total.

Variants, including the control is how many groups traffic is split between. Two means a control and one challenger.

Visitors a day entering the test is the field that goes wrong most often. It is not the site's daily traffic; it is the number of people who reach the thing being tested and are eligible to be split. A test on a product page takes visitors to that page. A test on a checkout step takes people who reach that step, which can be a small fraction of the site. Using the site-wide figure produces a duration several times shorter than the truth and a plan nobody can keep.

Visitors needed in total is the per-variant sample times the number of variants — the traffic the whole test consumes.

Days to run, rounded up, is that total over the daily traffic. Weeks is the same figure divided by seven and deliberately left fractional: the useful reading of 2.29 weeks is that it does not fit into two.

Visitors a day per variant is the figure that tells you whether the daily numbers will be readable at all. At 900 a day per variant with a 3 per cent rate, each variant is collecting about 27 conversions a day, and a day that lands at 20 or 35 is ordinary noise rather than news.

The single most useful rule on this page is to run in complete weeks, even when the sample arrives sooner.

Traffic is not the same on a Tuesday as on a Sunday: the people differ, their intent differs, and the conversion rate differs with them. A test that covers ten days has covered one weekend and one and a half working weeks, so the variants have been shown to different mixes of those populations. The test will report that difference as a result, and it will look exactly like a real one.

Sixteen days is therefore a plan for twenty-one. The extra five days cost a little traffic and remove an entire class of false finding, which is the best trade available anywhere in testing.

The same logic applies to the start. A test begun on a Thursday and run for exactly two weeks ends on a Thursday, which is tidy; one begun the day a sale starts is measuring the sale. Choose a start that sits inside an ordinary stretch of the calendar, and if an unusual period is unavoidable, plan for it to sit in the middle of the window rather than at one end where it lands on one variant's early traffic and the other's late.

The number this page produces is often uncomfortable, and the discomfort is informative.

Beyond about six weeks, the assumption that nothing else changed stops holding. Seasonality arrives. Creative fatigues. Prices move. A site release lands. A competitor starts bidding. Each of those affects both variants, which is meant to be the protection — but it also changes the underlying rate, and a test whose baseline drifted is measuring two things at once.

Long tests also cost the thing they were meant to protect: the decision. A result that arrives in August about a change proposed in June is frequently irrelevant by the time it lands, and the team has spent two months not shipping either version.

When the duration is unacceptable, the responses are the same four the sample size page gives. Test a bigger change, test something higher up the funnel where the rates are larger, accept a lower confidence level knowingly, or make the change on judgement and keep it reversible.

Adding a third variant does three things at once, and only the first is obvious.

It raises the total traffic needed by half. It splits the daily traffic three ways, so each variant fills more slowly than the arithmetic alone suggests. And it multiplies the number of comparisons, which raises the chance that one of them crosses the line by accident — properly handled, that needs a stricter threshold, which needs a larger sample again.

Two at a time, run in sequence, is usually faster than four at once and always easier to read.

Write the end date down before the first visitor enters, and treat it as the plan rather than as a suggestion.

The reason is not discipline for its own sake. A test read continuously and stopped at the first favourable moment produces far more false winners than its confidence level implies, because every look is another opportunity for noise to cross the threshold. The sample was sized for one analysis; taking twenty is a different procedure with a different error rate.

If continuous monitoring is genuinely needed — and sometimes it is, because a variant can be actively losing money — use a method built for it, and accept that it requires more data for the same confidence. What does not work is sizing a test one way and reading it the other, which is the default behaviour of almost every testing dashboard ever built.