What is Incrementality?

A holdout splits an audience in two. The treated 200,000 people see the campaign and produce 4,800 conversions, a rate of 2.40%. The held-out 200,000 see nothing and produce 3,900, a rate of 1.95%. Incremental lift is (2.40 − 1.95) ÷ 1.95 = 23.1%, and the campaign caused 4,800 − 3,900 = 900 conversions. The platform claimed 4,100 of the 4,800, so roughly 900 ÷ 4,100 = 22% of what it claimed was incremental.

Incrementality is the portion of results that would not have happened without the advertising.

Every other measurement in an account describes what occurred: impressions served, clicks received, conversions credited. Incrementality asks about something that did not occur — the sales the business would have made anyway — and that is not in any dataset because it never happened.

The only way to obtain it is to create a comparison. Withhold the advertising from a group comparable to the one receiving it, and measure the difference between them.

Incremental lift = (treatment conversion rate − control conversion rate) ÷ control conversion rate

The arithmetic is straightforward. The design is where the work is.

Attribution credits advertising that was present when a conversion happened. Presence and cause coincide sometimes and diverge systematically in particular places.

Branded search. Somebody who has decided to buy types the business's name, sees the ad above the organic listing, clicks it and buys. The conversion is credited, the money is real, and the advertising may have changed nothing but which link was used.

Retargeting. The audience is people who already visited and showed interest. A substantial share intended to return.

Any warm audience. Existing customers, recent visitors, people in an active purchase process — all of them convert at rates that have little to do with whether an ad was served.

Because these are also the cheapest conversions to acquire, they report the best numbers in the account. The gap between credit and cause is therefore widest exactly where the reporting looks strongest, which is the single most important thing to know about performance measurement.

A holdout. Split an audience at random, advertise to one half, withhold from the other, compare. The cleanest design available, and it requires a platform that supports randomised suppression at the audience level.

A geographic split. Match regions on prior performance, run the campaign in some and not others, compare. Available to everybody, noisier because regions differ in ways matching cannot fully control, and the right tool where audience-level holdouts are not offered.

A clean on-off. Stop the activity entirely for a defined period and compare against the period before. The weakest design, because seasonality, competitors and everything else move at the same time, and sometimes the only one available.

A platform lift study. Several platforms offer to run a holdout internally and report the result. Convenient, and the party running the test is the party being measured, which is worth weighing.

The most common way an incrementality test fails is that it was never capable of producing an answer.

A test detects an effect only if it has enough conversions for the difference to stand out from ordinary variation. A campaign producing a handful of conversions a week cannot detect a modest lift no matter how long it runs, and the result — no significant difference — gets read as evidence the advertising does nothing.

The sizing is done in advance: decide the smallest lift worth acting on, and calculate the conversions per group required to detect it. If the account cannot supply them in a reasonable period, the honest conclusion is that this test is not available, not that the campaign was tested and failed.

Change nothing during the test. A creative refresh, a budget change, a bid adjustment or a structural edit inside the window confounds the comparison beyond repair.

Cover the purchase cycle. Conversions arrive after the window if the window is shorter than the time people take, and cutting it short undercounts the treated group.

Watch for contamination. People move between geographies, share devices and see advertising through other channels. A control group exposed to the campaign through another route is not a control group.

Accept the cost. The held-out group produces fewer conversions than it would have. That forgone revenue is the price of the answer, and it is usually small against the budget the answer will redirect.

The output is a lift figure and a count of incremental conversions. Two comparisons make it useful.

Against what the platform claimed. If the campaign claimed 4,100 conversions and caused 900, roughly a fifth of the credit was incremental. That ratio, applied to the campaign's reported cost per acquisition, gives an incremental cost per acquisition — which is the number to compare against the business's ceiling.

Against the alternative use of the money. A campaign with low incrementality is not necessarily one to stop; it is one whose budget might work harder elsewhere, and the comparison needs both figures.

The answer is not permanent. Creative changes, audiences shift, competitors enter and leave, and the share of a campaign's results that are incremental moves with all of it.

Testing once and citing the result for years is a common habit and a poor one. A schedule — the largest budget lines annually, anything being argued about before the argument — keeps the steering honest without turning the account into a laboratory.

A test produces one number for one campaign over one period, and an account makes decisions every week. The bridge between them is a ratio rather than a figure.

If a campaign claimed 4,100 conversions and caused 900, the account now knows that roughly a fifth of that campaign's attributed credit is real. Applying that ratio to the campaign's ongoing reported cost per acquisition produces an incremental cost per acquisition that can be compared against the ceiling contribution allows.

The ratio holds for a while and not forever, which is why the cadence matters. It also differs sharply between campaigns — a prospecting campaign's ratio is typically far higher than a retargeting campaign's — so it is applied per campaign rather than as an account-wide correction.

Accounts that carry a small table of these ratios beside their attributed reporting make visibly better budget decisions than accounts that do not, and building it requires two or three tests rather than a measurement programme.

Long-term effects. A test measures the window it ran in. Advertising that builds recognition over years is undermeasured by any short test, which is a real limitation and not a reason to dismiss the method.

Channel interactions. Withholding one channel can change what another produces, so a test of one campaign in isolation attributes some of that interaction to the campaign under test.

Whether the alternative is better. A campaign shown to be incremental may still be a worse use of the money than something untested.

Attribution assigns credit for a conversion that happened; incrementality asks whether it would have happened anyway. That is a difference of question rather than of method, and only one design can answer the second: withhold the advertising from a comparable group and measure what changes. Everything that makes incrementality awkward follows from that. It is slow, because a test has to run for a purchase cycle. It is coarse, because it answers for a campaign or a channel over a period rather than for a keyword on a Tuesday. It costs money, because the held-out group converts less. And it is the only number in the panel that survives a finance review, because it is the only one that measures what the advertising caused rather than what it was present for. The practical arrangement is not to choose: attribution steers day to day, and a test run periodically on the largest lines says whether the steering has been pointing anywhere real.

• Test the campaigns whose attributed numbers look best, not the ones that look worst. The gap between credit and cause is widest exactly where the reported performance is strongest.
• Size the test before running it. A lift too small for the sample to detect produces an inconclusive result that gets read as a null one, which is the most expensive outcome available.
• Hold the test still. A campaign edited mid-flight, a budget change or a seasonal event inside the window all confound the comparison beyond repair.
• Exclude the people who were going to buy anyway where the design allows it, because a test run against a warm audience measures mostly coincidence.
• Re-run on a cadence rather than once. The answer moves as the creative, the audience, the mix and the market change, and a result from two years ago is a fact about two years ago.