Size a test before you run it

Your current rate. Whatever the test is about: the share of people who click, the share who convert, the cost per result. Read it from your own account over a window long enough to be stable, not from a benchmark. There is no published figure for your audience and your offer, and a guide quoting one would be quoting something it cannot source.

The difference worth acting on. Not the difference you hope for — the smallest one that would actually change what you do. If a five per cent improvement would not change a single decision, sizing the test to detect five per cent is buying precision you will not use.

The volume you can afford. Impressions, clicks or conversions per side, within the window and the budget you have.

The test is worth running when the volume you can afford is at least the volume the first two numbers require. That comparison is the whole exercise.

The volume required rises steeply as the difference you want to detect shrinks. Halving the difference does not double the volume; it multiplies it several times over.

That single fact explains most of what goes wrong in creative testing. Small accounts test for small improvements, conclude something on a handful of conversions, and act on a result that would have gone the other way on a different fortnight. The problem is not the statistics; it is that the test was never capable of answering.

It also explains why the biggest swings are the ones worth testing. A different offer, a different format, a fundamentally different hook — the differences large enough to be detectable on a modest account are exactly the ones large enough to matter.

The trap here is real and common: a test that looks well-powered on clicks and is hopeless on conversions.

Clicks are plentiful, conversions are not, and the conversion count is almost always the binding number because it is what the decision is about. If you will change budgets on the strength of cost per acquisition, size on acquisitions.

Work the front half on your own figures rather than on a rule of thumb. The impressions-needed calculator turns a target number of results into the impressions behind it, the CTR calculator and the CPA calculator take your own counts, and break-even CPA gives you the line the winner has to clear for any of it to matter.

Volume per side, divided by the volume the account produces per day, is the number of days. Two things make that estimate optimistic if you let them.

Delivery is not flat. A campaign underspends its nominal rate for ordinary reasons, and the first days of anything new are spent with the delivery system working out who to show it to. Those days count towards the window and contribute less towards the volume.

A budget cannot simply be raised to compress the window. A budget may not move more than a hundred per cent of what the entity carries today, applied increases on a connection are summed over a rolling twenty-four hours against the account's own daily ceiling, and that ceiling is derived from what the account has actually done — three times the highest single-campaign daily spend in its last twenty-eight days, floored at a default in the account's own currency. An account with a quiet month cannot buy a fast test.

So the window is usually longer than the naive division suggests, and it is better to find that out now than in week three.

Two lines, before anything starts running: the date or volume at which you will read the result, and what difference would count as a finding rather than noise.

If you want an early stopping condition, decide it in advance and make it stricter than the final one. Peeking at a running test and stopping when it looks good is how noise gets promoted to a finding — an early lead in a small sample reverses often enough to be worthless.

And write the prediction. Which side you expect to win and by roughly how much. It costs a line and it is the difference between a finding and a story assembled afterwards.

Four honest responses, in rough order of preference.

Test a bigger difference. Change something substantial rather than something incremental. This is usually the right answer and it is usually also the more interesting test.

Run it longer. If the window is tolerable and the account is stable, time is the cheapest input you have.

Accept the decision on judgement. Some decisions are not worth measuring. Making that call explicitly is very different from making it implicitly by running an under-powered test.

Do not run it. A budget spent on a test that cannot answer is a budget spent on nothing, and the finding will be acted on anyway, which is the part that costs money later.

The arrangement any of the resulting changes sit inside, stated once: your connections read your accounts, a change is written as a draft on the record with the state it would change from and the checks it passed, applying it is gated on approval — by you, or by a rule you wrote — and no platform write reaches a customer account until that platform has granted write access. Applying a cleared change to a platform is in private beta.

Run a creative test that answers something is the design the sizing serves, and the test plan template has a field for each of the three numbers so they get written down rather than estimated in somebody's head.