Test plan template for Fintech

The test plan with three columns a regulated financial product needs: which product the test runs against, which claim the variant is putting under test, and the review reference that cleared both arms of it.

Everything else is the general test plan, unchanged. One hypothesis with a because in it, one variable, a named control, a minimum run, a read date set in advance and a decision rule for every outcome.

Use it when a change to wording has to be cleared before it can be tried, which in this category is most of them. Naming the claim in the row is what lets the review happen once, before the build, rather than as a surprise after it.

Use it again to keep a record of which claims have been tested and which were refused, so the same idea is not proposed every year.

In most categories a test begins with an idea and ends with a build. Here it begins with an idea, goes through a review, and only then becomes a build. A plan that does not reflect that order will keep producing tests that are finished, unreviewable and stuck.

The claim under test column is what fixes it. Write the claim the variant makes in plain words — a fee position, a speed, an eligibility, a comparison — and send that sentence to review before anybody designs anything. Reviewing a sentence takes an hour. Reviewing a finished campaign takes a round of rework, and the rework happens after the test was supposed to start.

Both arms need a reference, not just the variant. The control is usually already cleared, but a test that changes where a claim sits, or what surrounds it, can change what the control is claiming too.

When a claim is refused, the row does not disappear. It is recorded as refused, with the date and the reason.

That is a genuine finding about the product, and it is the kind most often lost. Without the record, the same idea is proposed every year by somebody new, spends another review cycle, and is refused again for the same reason. With it, a marketer can read what this product has been allowed to say and what it has not, which is the most practical guide available anywhere.

Keep those rows alongside the ones that ran. A test plan in this category is partly a test record and partly a claims history, and the second half is the one that compounds.

The most expensive mistake available here is testing on applications.

A variant can reliably lower the cost per application by promising more, or by reaching people who are further from qualifying. The applications arrive, the cost looks excellent, and the approval rate falls by more than the cost improved. In a plan that reads on applications a week after launch, that is a clear win and it will be adopted across the account.

So the primary metric should be a funded account or an approved application, and the read date has to allow time for those to land. Tests here are slower than in other categories for a structural reason, and shortening them does not make them faster, it makes them wrong.

Where the hypothesis is specifically about applicant quality, approval rate can be the metric. Be careful with volume: a rate calculated from a small number of applications moves enough to justify any conclusion.

The product column exists because a finding rarely travels between an account, a card and a loan.

The audiences differ, the approval rates differ, and the reasons somebody abandons an application differ. A hook that works for a low-commitment product may perform badly for one that requires a credit decision, and the reverse is also true. Recording the product in the row is what stops a single good result being rolled out across a portfolio where it does not apply.

The landing page is a frequent invisible variable here, because a claim usually has an obligation attached to it.

If the variant's claim requires a different disclosure, the page changes, and the test is then measuring the page as much as the ad. Sometimes that is the intended test. Often it is not, and nobody noticed.

Write in the notes which page each arm points at and what disclosure it carries. Where the pages differ, say so plainly and treat the row as a test of the pair rather than of the headline.

Where a test includes a retargeting arm, note that the audience is made of people who began an application, which is an audience defined by a financial circumstance.

What may be said to it is narrower than what may be said to a cold audience, which means a variant cleared for prospecting is not automatically cleared here. Give the retargeting arm its own review reference, or keep it out of the test.

Application volumes in this category are often large enough to test properly, which is a genuine advantage over most regulated industries. Funded-account volumes usually are not.

That gap decides what is worth a row. Tests whose effect should be visible in applications — a headline, a landing page, an audience — can be read reasonably quickly. Tests whose effect is on who qualifies, which is the more valuable question, need a longer run and a larger sample, and shortening them produces the confident nonsense this plan exists to avoid.

The practical rule is to test fewer, larger things and to give them time. A plan with four well-run rows a quarter teaches more than one with twenty that were each read too early.

Demand for a financial product moves with things that have nothing to do with advertising: a rate announcement, a competitor's promotion, the end of a financial year, a news story about the category.

Run both arms simultaneously so that whatever moves in the market moves under both, and write into the notes anything that happened during the run. At the read date those events are the alternative explanations for the result, and nobody will recall them a month later.

Sequential tests, where the variant runs after the control, should be treated as changes rather than tests. Too much sits in the gap between them.

Every row has a hypothesis with a because in it. Every row names the claim the variant makes, in plain words, and both arms carry a review reference with a date. The primary metric is a funded account or an approved application. The read date allows those to land. The decision rule covers the outcome where nothing moves. And refused claims are recorded as rows too, with the date and the reason, because that record is the most useful half of this document.