Test honestly, substantiate, and sign off
How do you find out what works without deceiving the people you test on, and who signs off?
This chapter covers A/B testing with the metric, sample size, and stopping rule fixed in advance, explained without statistics jargon. It then walks through claim substantiation, legal review, the front-page test, and a sign-off record with named people who can stop the launch.
All stages
Decide the metric and the rule before the test
A test is only informative if you decide what counts as winning before you see any data. Write a one-page test plan: the hypothesis, the single primary metric (it should be the behavior in your objective, such as plan starts per visitor, not clicks), the guardrail metrics from stage 1, and the decision each possible result leads to. Date it and file it.
Every variant you test must already have passed the gates for message, channels, and creative. You are comparing one honest version with another, never an honest version with a deceptive one to see how much the deception is worth. People in a test are customers, not subjects: assign them at random, and do not give one group worse prices or terms than they would accept if they knew about the other.
Sample size and stopping, in plain language
Small differences need many people. A workable rule of thumb for each variant is sixteen times p times (1 minus p), divided by the square of d, where p is your current conversion rate and d is the smallest difference worth acting on, both written as decimals. With a 3 percent baseline and a target of 3.75 percent, that comes to about 8,300 visitors per variant. If you do not have that traffic, test a bolder difference, or accept that the test cannot settle the question and decide on judgment, saying so.
Fix the sample size or the end date in advance, run whole weeks so that weekdays and weekends are both covered, and do not stop because one version is ahead. Checking daily and stopping at the first apparent win produces false winners far more often than the stated error rate suggests. Stop early for one reason only: a guardrail breach that suggests customers are being harmed. Report every result, including the dull and the losing ones.
Review claims and substantiation
Before sign-off, someone who did not write the copy checks each claim in the launch version against its evidence in the ledger. The evidence has to match the claim as worded: "within 48 hours" must be true of substantially every shipment, not of a good week. Where it falls short, narrow the claim.
Send regulated and high-risk claims for legal review: health or wellness benefits, environmental claims, comparisons with named competitors, "free", guarantees, and origin. Health-related claims need competent and reliable scientific evidence, which usually means good-quality human studies relevant to the product, not a supplier leaflet. For subscriptions, have the terms checked: clear disclosure before payment details are taken, express consent to the recurring charge, and a simple way to cancel.
Run the front-page test and sign off
Give the whole package (brief, audience profile, creative, test results) to a colleague whose job is to find fault, and answer their objections in writing. Then write the most critical accurate headline a journalist could publish with full access to the same material. If that headline embarrasses you, change the campaign, not the headline.
Record sign-off by name and date: claims, legal, accessibility, and support readiness. Each signatory can stop the launch, and the record says which version they approved.
Techniques at this stage, and where the line is
When evidence is partial, qualify the claim so that it says exactly what the evidence shows, and keep the qualifier as visible as the claim.
The line: Qualifiers such as "up to" or "may help" used so the reader still takes away the strong claim you could not prove.
Leave open only what is truly undecided, and say that it is undecided.
The line: Wording chosen because customers will read it as a firmer promise than you are prepared to keep.
You cannot see what a newcomer will misread. Test comprehension with people outside the team: ask them what they think they will be charged, and when.
The line: Knowing that readers misunderstand the offer in your favour and leaving the wording as it is.
Trust differences only from randomised comparisons that reached their planned size, not patterns spotted in a dashboard.
The line: Presenting a chance pattern as proof to colleagues, investors, or customers ("proven to convert" or "customers prefer").
Check that an overall result is not produced by a different mix of visitors in each arm; compare like with like across devices and traffic sources.
The line: Choosing whichever level of aggregation gives the answer you wanted and reporting only that.
Most tested ideas make little or no difference, so treat a surprising win with suspicion and repeat the test before building on it.
The line: Advertising a result from one small test as an established fact about the product.
Worked example
Kestrel Coffee has two honest headlines for Flex and one claim still marked unverified. During a three-week soft opening before the 12 January launch, about 800 visitors a day reach the Flex page.
- 1The test plan is written and dated before any traffic is split. Hypothesis: "Stop coffee piling up" (B) will beat "Coffee that waits for you" (A). Primary metric: Flex starts per visitor. Guardrails: refund requests and complaint contacts. Rule: adopt B only if it wins on the primary metric and neither guardrail is worse.
- 2Sample size: with a 3.0 percent baseline and 3.75 percent as the smallest lift worth acting on, the rule of thumb gives about 8,300 visitors per variant, which is 21 days at current traffic. Nobody acts on the dashboard before day 21.
- 3On day 9 version A is ahead and the founder wants to stop and keep it. The plan holds. At day 21, A has converted 3.0 percent and B 3.8 percent, with refund requests and complaints level in both arms. B becomes the launch headline.
- 4Substantiation: ninety days of roast logs show 91 percent of bags shipped within 48 hours of roasting and every bag within five days. "Roasted within 48 hours" is not true of substantially every shipment, so it is replaced with "roasted the week it ships", which the logs support.
- 5A lawyer on a fixed-fee review checks the subscription terms: the recurring price is disclosed before card details are taken, consent to the recurring charge is a separate unticked box, and cancellation works as described.
- 6Front-page test: the support lead writes the harshest accurate headline she can: "Local roaster's new plan: half-price first bag, then 17 dollars a bag, cancel in two clicks." Nobody objects to seeing it in print.
Outcome. The launch version is fixed with headline B and the softened roast claim. The test plan, full results, and a sign-off sheet with three names (founder, support lead, lawyer for the terms) are filed a week before launch.
Test plan and sign-off record
Fill in the first four fields before the test starts and the rest before launch. Saved in this browser only.
What you expect to happen and why, naming both variants.
The one metric that decides the test, plus the guardrail metrics that must not get worse.
Baseline rate, smallest difference worth acting on, visitors needed per variant, and how many whole weeks that takes.
Choose the rule you committed to before the test began.
Each launch claim, the evidence checked, the name of the reviewer who did not write it, and any claim narrowed as a result.
Tick each area present in the campaign that a qualified reviewer has checked.
The most critical accurate headline a journalist could write about this campaign with full access to your files.
Name, role, date, and version approved for claims, legal, accessibility, and support readiness.
Ethics gate: testing and approval
0/8If any of these is not true: Do not launch. Rerun a test that was stopped early or had no plan, and treat its earlier result as unknown. Narrow or remove any claim that failed review. If the front-page headline is embarrassing, trace it back to the stage that produced the problem and redo that stage before returning here.
Common mistakes
- ×Watching the dashboard daily and stopping the moment your preferred version pulls ahead.
- ×Choosing the metric after the results are in, or slicing by segment until something looks like a win.
- ×Testing a misleading variant "just to see", which means some real customers were misled.
- ×Optimising for clicks, which rewards the headline that over-promises.
- ×Treating legal clearance as the whole review: a claim can be lawful and still leave customers feeling tricked.
Am I part of an experiment, and how would I tell?
Related
References
- Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University PressChoosing metrics in advance, guardrail metrics, and common pitfalls in online experiments.
- Kohavi, R., Longbotham, R., Sommerfield, D., & Henne, R. M. (2009). Controlled experiments on the web: Survey and practical guide. Data Mining and Knowledge Discovery, 18(1)The sample-size rule of thumb: about sixteen times the variance divided by the square of the smallest difference worth detecting, per variant, for 80 percent power at the 5 percent level.
- Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11)Flexible stopping and metric choice inflate false-positive results.
- Johari, R., Koomen, P., Pekelis, L., & Walsh, D. (2017). Peeking at A/B tests: Why it matters, and what to do about it. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data MiningRepeatedly checking a running test and stopping on significance inflates the error rate.
- Federal Trade Commission (2022). Health Products Compliance GuidanceHealth-related claims require competent and reliable scientific evidence matched to the claim as consumers understand it.
- Federal Trade Commission (1984). FTC Policy Statement Regarding Advertising SubstantiationA reasonable basis for objective claims must exist before they are published.
- Baker, S., & Martinson, D. L. (2001). The TARES test: Five principles for ethical persuasion. Journal of Mass Media Ethics, 16(2-3)The five-part test applied at sign-off.