Test with informed participants, then put every product through an approval gate
How do you find out what works without deceiving the people you test on, and who signs off?
This chapter shows how to find out what works without deceiving anyone: pre-testing with people who know they are testing, metrics fixed in advance, and a red team invited to break the plan. It then sets the approval gate: the TARES test, the front-page test, a named approver who did not write the products, and legal review of disclaimers.
All stages
Pre-test with people who know they are testing
Doctrine separates making a product from releasing it: phase five exists so that nothing goes out until someone with authority, who did not write it, has approved it. Before approval, find out whether the products work, and do it with people who know they are helping you test. Recruit participants from the audience, tell them who you are, who is paying, and what the test is for, pay them for their time, and let them stop at any point. A dozen interviews in which people read a product aloud and tell you what it means will catch most problems.
If you compare versions in a live setting, every version must already be truthful and attributed, and the comparison must be one you would describe to the audience without discomfort. Experiments that vary how much pressure or distress people are exposed to, without their knowledge, are not testing. They are the harm.
Choose the metrics before you look
Write down what you will measure, and what result would make you change or drop a product, before you see any data. Measure understanding first: can people say what the product asks, where, when, and at what cost? Then measure what they intend to do, whether they felt pressured or talked down to, whether they could tell who it came from, and how they now feel about people who disagree.
Look for backfire in subgroups. In a randomised trial of vaccine messages, Nyhan and colleagues found that corrective content reduced a misperception and yet lowered intention to vaccinate among the parents with the least favourable attitudes. A test that reports only the average, or only the metric that came out well, is a test designed to approve.
Red-team the whole package
Give everything to people whose job is to break it. A red team needs at least one sceptical member of the audience, one person from a group that could be hurt by the campaign, and one person who understands the law. Ask them four questions. What is the most misleading reading of this? Which claim is weakest? Who could this harm, including people outside the audience? What would a fair critic say on seeing the full plan, the targeting, and the budget? Record every objection and what you did about it.
The approval gate
Approval is a decision by a named person who did not create the products. Two tests make the decision concrete. The TARES test (Baker and Martinson) asks whether the message is truthful, whether the persuader is authentic, whether the audience is respected, whether the appeal is equitable given what the audience knows and can check, and whether the campaign is socially responsible. The front-page test asks whether you would be content to see the complete plan (messages, targeting, messengers, payments, and funders) reported accurately in the newspaper your audience reads. A product passes both or it does not go out.
Legal review
Have counsel check the attribution and paid-for-by lines, the disclosure wording for anyone compensated, consent for every contact list and for text messaging, image and quote releases, and any rules specific to your subject. If the campaign touches an election or a ballot question, campaign-finance registration, reporting, and disclaimer requirements apply and vary by jurisdiction; get that advice in writing before launch. Legal clearance is a floor. It does not replace the two tests above.
Techniques at this stage, and where the line is
Ask the red team to state the strongest case against the campaign, and make the plan answer that case.
The line: Showing the approver a weakened version of the critics' concerns, so that sign-off is easy, manipulates your own review.
Let participants criticise freely: anonymous answer sheets, and a moderator who did not make the products.
The line: Treating polite answers as evidence, or recruiting participants likely to approve, produces a result you arranged.
Show drafts openly, as drafts, under your own name, to the audience and partners in order to learn from the reaction.
The line: Floating a claim anonymously to see whether you can get away with it, while keeping deniability, is the manipulative form.
There is no honest use. Honest message testing is consented research, described to participants as research.
The line: Advocacy delivered in the guise of a survey deceives the respondent about what is happening to them.
Guard against it in approval: an independent approver, outside reviewers, and dissent written into the record.
The line: Staging a review whose members were chosen because they would agree is a performance of oversight.
If software drafts message variants, each variant is held to the same truth and attribution standard, tested with informed participants, and labelled where required.
The line: Automated tailoring of arguments to an individual's inferred psychological traits, without their knowledge, crosses into covert profiling.
Check subgroups for reversals, and pull any product that hardens the people it was meant to help.
The line: Releasing a product you know inflames or entrenches one group, because the average looks good, trades their harm for your metric.
Worked example
Fictional example, continued. The Riverside Health Partnership has five products. The approving authority is the county health officer, Dr Lena Brandt, who has seen none of the drafts.
- 1Metrics are written, dated, and shared with partners before testing: at least 80 percent of readers can state where, when, and the cost after one reading; no more than 10 percent report feeling pressured or judged; no fall in stated intent among parents with questions. A product that misses any of these is rewritten or dropped.
- 2Pre-test: 36 parents recruited through schools and the parish are told who is testing and why, paid 20 dollars each, and answer on anonymous sheets. A researcher from the community college, who is not on the team, moderates.
- 3Results: 31 of 36 (86 percent) can state where, when, and the cost from the rewritten flyer. The line reading: do not let your child be the one sent home, fails. Nine of 36 (25 percent) say it feels like a threat, and intent falls among the eight parents who had questions.
- 4Red team: a parent who has decided against vaccinating, a tenant advocate from North Quay, and county counsel. They find that the radio spot implies every vaccine is given in one visit, and that the registry-based letter does not explain how the family was identified.
- 5Changes: the sent-home line is removed, the radio spot is corrected, and the letter now says why the family is receiving it and how to ask questions or stop further letters.
- 6Approval: Dr Brandt applies the TARES test and the front-page test to each product with the red-team log beside her. She approves four and returns the video for a caption correction. Counsel signs off the paid-for-by lines and the voucher disclosure in writing.
Outcome. A signed approval sheet for each product, the red-team log, and the metrics written in advance, all filed for publication with the final report.
Testing and approval record
Keep one record for the campaign and attach an approval line for each product. Saved in this browser only.
Written and dated before any data is seen, including the result that would make you rewrite or drop a product.
Tick what is true of every participant.
Who reviewed, and their answers on misleading readings, weak claims, possible harm, and the fair critic's view.
Each objection and what you did about it, including products dropped.
The approver ticks each principle the product satisfies.
Imagine the full plan, including targeting and funders, reported accurately in the audience's own newspaper.
Tick each item counsel has checked.
A named person with authority who did not create the products, and the date of the decision.
Ethics gate: Phase V, approval
0/7If any of these is not true: Do not launch. Fix the product and test it again, or drop it. If the approver or counsel has not reviewed it, wait; a deadline is not a reason to skip the gate.
Common mistakes
- ×Choosing the success metric after seeing which one came out best.
- ×Testing on colleagues and friends, who know too much and are too kind.
- ×A red team made up of people who already agree, given one afternoon.
- ×Letting the author approve their own work, or treating approval as a formality.
- ×Treating legal clearance as ethical clearance.
Am I part of an experiment, and how would I tell?
Related
References
- Department of the Army (2013). FM 3-53: Military Information Support OperationsApproval as a separate phase: no product is released without sign-off by a designated authority.
- Baker, S., & Martinson, D. L. (2001). The TARES test: Five principles for ethical persuasion. Journal of Mass Media EthicsThe five-principle test used here as the approval gate.
- Nyhan, B., Reifler, J., Richey, S., & Freed, G. L. (2014). Effective messages in vaccine promotion: A randomized trial. PediatricsWell-meant corrective messages can reduce a misperception while lowering intent in the most hesitant subgroup, so subgroup effects must be tested.
- Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of SciencesFixing outcomes and analyses in advance separates prediction from after-the-fact explanation.
- National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research (1979). The Belmont Report: Ethical Principles and Guidelines for the Protection of Human Subjects of ResearchInformed consent and respect for persons as conditions for research involving people.
- Voelkel, J. G., Stagnaro, M. N., Chu, J. Y., et al. (2024). Megastudy testing 25 treatments to reduce antidemocratic attitudes and partisan animosity. ScienceComparing many candidate interventions against outcomes fixed in advance shows that plausible ideas differ widely in effect.