What is Holdout test?

A holdout test splits an audience at random. One part is advertised to, the other is not, and the difference in outcomes between them is what the advertising caused.

The design is the standard one for establishing causation in any field, and its virtue is the random assignment rather than the withholding. Randomisation makes the two groups comparable on everything — including the characteristics nobody thought to control for — so that the only systematic difference between them is the treatment.

That is what a geographic test cannot fully achieve. Matching regions on observable history is a good approximation and it relies on the unobservable differences being small. Random assignment removes the assumption.

The most common way this test fails is that it was never capable of producing an answer, and that failure is entirely avoidable because the calculation is done in advance.

Three inputs decide the size.

The baseline conversion rate of the audience, which is known from the account's own history.

The smallest lift worth acting on. Not the lift hoped for — the smallest one that would change a decision. A test designed to detect a large effect will miss a moderate one and report nothing.

The confidence required, which is a judgement about how wrong the business is willing to be and in which direction.

From those comes the number of people, or more usefully the number of conversions, needed on each side. If the audience cannot supply them in a reasonable period, the honest conclusion is that this test is not available for this campaign — not that the campaign was tested and found wanting.

Getting this backwards, choosing a holdout share by feel and discovering afterwards that nothing could have been detected, is how accounts end up with a drawer of inconclusive tests and no change in behaviour.

Test the campaigns reporting the best numbers.

This is counterintuitive and it is where the money is. Retargeting, branded search and any campaign serving a warm audience collect attributed credit efficiently, because they advertise to people already close to buying. Their reported cost per acquisition is genuinely low and substantially overstates what they caused.

Testing the campaign that already looks poor generally confirms that it looks poor. Testing the one that looks excellent is where accounts discover that a fifth of the credit was real, and where a budget decision worth making becomes visible.

Suppress properly. The held-out group must not receive the campaign through another route — a lookalike built from the same seed, a second campaign with an overlapping audience, a retargeting pool that includes them. Audience overlap is the most common source of contamination and it is invisible unless somebody checks.

Change nothing during the window. Creative refreshes, budget changes, bid adjustments and structural edits all confound the comparison.

Run for the full purchase cycle. Conversions arriving after the window are conversions the treated side does not get credit for, which biases the result toward finding no effect.

Measure from the system of record. Orders, customers and revenue from the business's own data, split by group. Reading the ad platform's attributed figures measures the thing the test exists to check.

Do not peek and stop. Watching a test and stopping when the difference looks convincing inflates the chance of a false positive substantially. The stopping point belongs in the design.

The held-out group produces fewer conversions than it would have. That forgone revenue is the price of the answer.

It is usually small against what the answer is worth. A test costing a few thousand dollars of forgone conversions that redirects a six-figure annual budget has paid for itself many times over, and the calculation is worth doing explicitly when somebody objects to the cost — which somebody always does.

Where the cost genuinely bites is on small audiences, because the holdout has to be a large share of a small pool to detect anything. That is a real constraint and an argument for geographic testing instead.

The output is a difference in conversion rate between the groups, a confidence interval around it, and a count of incremental conversions.

Two conversions matter for using it. Dividing spend by incremental conversions gives an incremental cost per acquisition, which is the figure to hold against the ceiling contribution allows. And comparing incremental conversions against what the platform claimed gives a ratio — the share of attributed credit that was real — which can be applied to the campaign's ongoing reporting between tests.

That ratio is the practical output. It is what turns a one-off experiment into something the account uses every week.

Rather than running a test and stopping, some accounts keep a permanent holdout: a small, randomly chosen share of the audience that never receives a particular campaign.

The advantages are considerable. The measurement is continuous rather than a snapshot, so the ratio between credit and cause can be tracked as creative, audiences and the market change. There is no test to design each time somebody asks. And a result is always available when a budget argument starts, rather than three weeks later.

The costs are the same forgone conversions, paid continuously rather than in a burst, and the operational discipline of keeping the suppression intact across every campaign change, audience rebuild and platform migration. That last one is where always-on holdouts quietly break: somebody duplicates a campaign, the exclusion does not come with it, and the control group has been advertised to for a month before anybody notices.

Where the discipline exists, it is the strongest measurement arrangement available to a performance account.

Several platforms offer to conduct a holdout internally and report the lift.

These are genuinely useful and worth using, with one thing held in mind: the party running the test is the party being measured, and the design, the analysis and the reporting are all theirs. The result is not independent evidence in the way a test the business ran on its own data is.

The practical stance is to use them for a first read, to prefer a design the business controls when the answer will move real money, and to be wary of a lift figure whose methodology cannot be inspected.