Test plan template for Ecommerce

The test plan with three columns a store needs: which slice of the catalogue the test runs against, which promotion window it sits inside, and what the variant is expected to do to returns.

Everything else is the general test plan, unchanged. One hypothesis with a because in it, one variable, a named control, a minimum run, a read date set in advance and a decision rule for every outcome.

Use it when a change to creative, offer or landing page is worth knowing about rather than merely shipping. In a store the same change can behave completely differently on a hero product and on clearance, so the segment the test ran against is part of the finding.

Use it again before a peak, because a test started inside a promotion is measuring the promotion as much as the variable.

A store does not have one audience and one product; it has a range, and the range is what varies. The same change — a new hook, a different first frame, a price on the creative — will often win decisively on a hero product and do nothing at all on the long tail.

That is why the catalogue segment column exists. A finding recorded without it reads as a general truth, gets applied across the whole account, and disappoints almost everywhere it is applied. A finding recorded with it reads as what it is: this worked on this kind of product, and the next question is whether it transfers.

It also changes how a test is designed. If the hypothesis is genuinely about the product rather than about the creative, the right test may be the same creative treatment across two segments rather than two treatments on one.

The promotion window column enforces a rule that is easy to state and constantly broken: a test cannot straddle a change of offer.

Retail offers are powerful. A discount, a free-returns period, a bundle or a delivery promise will usually move a conversion rate further than a creative variable will, and a test whose control ran at full price and whose variant ran inside a sale is measuring the sale.

The same applies to tests that begin in a promotion and end outside one. Where a test must run near a window, the honest options are to keep it entirely inside the window, keep it entirely outside, or accept that the row is a change rather than a test and mark it as one.

The most specific trap in this category is a variant that increases orders and increases returns by more.

It is easy to produce. Creative that overstates fit, urgency or condition will lift conversion and quietly raise the return rate, and in a plan that reads on orders a week after launch it looks like a clear win. The returns arrive later, against a cohort nobody is still looking at, and the finding stays in the record as a success.

So the returns impact column asks what the variant is expected to do to returns and how that will be checked. Where the variable could plausibly touch fit, expectation, sizing or urgency, the primary metric should be a kept order rather than an order, and the read date should allow enough time for returns to land against the same cohort.

This makes tests slower. It also makes them true, and in a category where return rates on some lines are substantial, a test plan that ignores them is a plan that can be won by making a worse ad.

A store has a confound that most categories do not: the thing being sold can run out.

A variant that performed well for six days and then lost its best-selling size on day seven has not been out-performed; it has been interrupted. A control pointed at a collection where three products went out of stock is not a fair comparison either.

Record stock events in the result column when they happen, and where a test is being run on a segment with thin cover, either choose a different segment or accept that the row will be inconclusive. There is no statistical fix for having nothing to sell.

A related confound is the calendar. Weeks around a delivery cut-off, a seasonal peak or a public holiday behave differently from ordinary weeks in ways that swamp most variables.

Where a test's minimum run has to cross one of those weeks, both arms cross it together and the comparison holds. Where a test starts inside such a week and ends outside it, the early days of the test carry a different world from the late ones, and the arm that happened to launch first gets a period the other one did not.

The simplest protection is to start both arms at the same moment and to avoid setting a read date in the week after a peak, when a store's numbers are least like its normal ones.

Not everything should be a row. Broken things are fixed, obvious improvements are shipped, and a plan cluttered with trivial rows makes the important ones harder to find.

In a store, the tests that repay the effort are usually about the offer, the first frame, the landing destination and the way a product is shown in use. Those four move results reliably enough that a finding about them is worth having. Colour, fonts and minor copy edits rarely produce a difference large enough to read against retail noise, and testing them consumes the traffic that a real test needed.

Every row has a hypothesis with a because in it. Every row names one variable, one catalogue segment and one promotion window, and no test straddles the edge of a promotion. Where the variant could touch returns, the primary metric is a kept order and the read date gives returns time to land. The decision rule covers the outcome where nothing moves. And every finished row has a result written on its read date, including the ones that were interrupted by stock.