# Test plan — Ecommerce

The test plan with three columns a store needs: which slice of the catalogue the test runs against, which promotion window it sits inside, and what the variant is expected to do to returns.

Everything else is the general test plan, unchanged. One hypothesis with a because in it, one variable, a named control, a minimum run, a read date set in advance and a decision rule for every outcome.

## How to use it

1. Copy this file, or the CSV version if you would rather work in a sheet.
2. Write the hypothesis before the variant, and make sure it contains a because.
3. Name one variable. If there are two, that is two tests.
4. Set the primary metric, the minimum run and the read date before anything is built.
5. Write the decision rule for every outcome, including the one where nothing moves.
6. Leave Result blank until the read date, and then fill it in whatever it says.
7. Keep the finished rows. A year of them is worth more than any single test in it.

## The columns

**Test ID** (Text). A short stable identifier that travels onto the creative file, the ad name and the result, so a finding can be traced back to the assets and the segment that produced it.

**Hypothesis** (Text). What you believe and why, with a because in it, and in a store the reason is usually about the product rather than the ad: this frame will hold viewers because the current open shows packaging rather than use.

**Variable** (Text). The single thing being changed. In a store the tempting mistake is to test a new creative that is also a new product, a new offer and a new landing page, which answers nothing.

**Control** (Text). What the variant is compared against, named exactly, including which colourway and which collection page. Six weeks later nobody remembers which version of the old creative was live.

**Variant** (Text). What is being tried, described so somebody could build it: the fit guide in the first frame, the offer removed from the image, the product shown in use rather than on a plain background.

**Audience** (Text). Who sees the test, identical on both sides. In a store the usual leak is a shopping campaign whose delivery differs between arms because the catalogue behind them is not the same.

**Where it runs** (Text). Which platform, campaign and placement carry this test. A test split across a feed and a shopping surface is two tests, because the two deliver against different signals.

**Primary metric** (Text). The one number this test is decided on, and where returns are material that is a kept order rather than an order. Chosen before the test starts, not after the first week's figures arrive.

**Minimum run** (Text). The shortest run before anybody looks, long enough to cover a full weekly cycle and, where returns matter, long enough for them to land against the same cohort.

**Read date** (Date (YYYY-MM-DD)). The day the result is read, set in advance, and never in the week after a peak, when a store's numbers are least like its ordinary ones.

**Decision rule** (Text). What each outcome means, written before the test starts, including the outcome where the two arms are indistinguishable, which is the most common one at retail volumes.

**Owner** (Text). The person who builds the test, reads it on the read date, and writes the result down whatever it says.

**Result** (Text). What happened and what was decided, written on the read date, including the stock event that interrupted an arm. Blank until then.

**Catalogue segment** (Text). The slice of the range the test runs against: best sellers, new arrivals, a hero product, clearance. The same change often behaves differently on each, so the segment is part of the finding.

**Promotion window** (Text). The promotion the test sits inside, or Full price. A test that straddles the edge of a promotion is measuring the offer change, not the variable.

**Returns impact** (Text). What the variant is expected to do to the return rate, and how that will be checked. A creative that raises orders and raises returns further has not won anything.

## The test plan

| Test ID | Hypothesis | Variable | Control | Variant | Audience | Where it runs | Primary metric | Minimum run | Read date | Decision rule | Owner | Result | Catalogue segment | Promotion window | Returns impact |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |
|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |   |

## Before you commit it

- Every row has a hypothesis with a because in it.
- Every row names one variable, one catalogue segment and one promotion window.
- No test straddles the edge of a promotion.
- The primary metric accounts for returns where the variant could plausibly move them.
- The read date allows enough time for returns to land.
- The decision rule covers the outcome where nothing moves.
- Every finished row has a result written on its read date.