Test plan template

A test plan is one row per test, and a test is one hypothesis with one variable in it. The row records what you believe and why, what is being changed, what it is being compared against, how long it runs, the day the result is read, and what each outcome will mean.

The last two columns are the ones that make it a test rather than a change. A read date set in advance stops a result being read when it looks best, and a decision rule written in advance stops the finding being reverse-engineered from whatever happened.

Use it whenever a change is worth making and worth knowing about. Not every change needs a test — a broken link is fixed, not tested — but anything that will be argued about later is cheaper to settle now with a row than in six months with recollections.

Use it again as a record. The completed rows are the thing that stops a team running the same test every year, and they are the only defence against a finding from three years ago that nobody can source.

The difference between testing and fiddling is one sentence. A test starts with a belief that has a reason in it: this will happen, because of that. A change starts with an idea.

The reason is what makes a losing test useful. Suppose a shorter opening cut loses. If the hypothesis was try a shorter video, all that has been learned is that one video did not work. If the hypothesis was a shorter opening will hold more viewers because the current cut spends its first three seconds on a logo, then losing tells you something about how your audience treats openings, and that generalises to the next twenty pieces of creative.

So the hypothesis column asks for a because, and rows without one should be sent back. It is the cheapest quality control in the document and the one most often skipped.

The variable column carries the whole logic of the test. Change one thing, hold everything else identical, and the difference in the result is attributable.

In practice, this is where most test plans quietly fail. A new creative is tested against the old one, but it is also a different length, on a different landing page, to a slightly different audience, because all four were improved at once. The variant wins, and nobody can say which of the four changes did it, so the finding cannot be applied anywhere else.

Two variables means two tests, run in sequence. Where speed genuinely matters more than knowing — a launch, a deadline, a promotion — shipping several changes together is a reasonable decision, but the row should say so plainly rather than claiming an attribution that does not exist. A test plan is more useful for containing a few rows marked as changes than for pretending everything in it was clean.

A test needs something to be compared against, and writing down what it is prevents a common and expensive drift.

Usually the control is whatever is already running. Name it exactly: this creative, this audience, this page, as it exists today. Six weeks later, when the result is being discussed, nobody will remember which version of the old creative was live at the time, and a control described as the current ads is a control that cannot be reconstructed.

The audience column carries the same burden. The audience has to be identical on both sides, and any difference is a second variable nobody wrote down. This sounds obvious and it is broken constantly, usually by a platform feature that optimises delivery differently between two arms of a test.

These two columns are what separate a test plan from a diary.

A minimum run is a commitment not to look. It can be a duration or a volume of results — long enough for the buying cycle, or enough conversions on both sides to tell a difference from noise. Whichever form it takes, the point is that it is agreed before anybody has seen a number.

The read date is when the decision happens. It exists because a test that is checked daily will be stopped on the day it looks best, and early results swing widely for reasons that have nothing to do with the variable. Every team that checks daily eventually finds a moment where the variant is ahead, stops there, and records a win. That is not a test, it is a search for a moment.

Where a test genuinely has to be stopped early — money is being wasted, something is broken — record that in the result. An early stop is a legitimate operational decision and an illegitimate finding.

Write what each outcome will mean, in advance, including the one where the two arms are indistinguishable.

That third case is the one almost every plan omits, and it is the most common outcome in practice. Without a rule for it, an inconclusive test gets resolved by whoever is most confident in the room, and the plan quietly becomes a way of dressing up opinions.

A reasonable rule reads: adopt if the variant leads on the primary metric, discard if it trails, run again with more weight if the two are within noise. It does not have to be sophisticated. It has to exist before the numbers do.

The result column is filled in on the read date, and it is filled in whatever it says.

No difference is a finding. It says the variable matters less than somebody believed, which saves the next person from spending a fortnight on it. A plan that only records winners produces an organisation that tests the same five things every year and remembers only the times they worked.

Keep the finished rows together. After a year they answer the question that no single test can: what has actually moved the number in this account, how reliably, and what has been tried and does not work. That record is worth more than any individual row in it, and it is the only defence against a finding from three years ago that everybody repeats and nobody can source.

Every row has a hypothesis with a because in it. Every row names exactly one variable, or says plainly that it is a change rather than a test. The control is named, not implied. The audience is identical on both sides. The primary metric, the minimum run and the read date were all set before anything was built. The decision rule covers the outcome where nothing moves. And the result column is empty until the read date, and never after it.