Measure whether the behaviour changed, and what the campaign cost the audience
Did the change you wanted happen, and what did it cost the audience?
This chapter separates what you did (measures of performance) from what changed (measures of effectiveness), and has you read the result against the baseline and the other causes you listed at the start. It also requires two things most reports leave out: honest null results, and harm monitoring, including whether the campaign raised contempt for people who disagree.
All stages
Performance is not effectiveness
Evaluation is the seventh phase of the cycle, and doctrine draws a line through it that most campaign reports blur. Measures of performance record what you did: letters sent, spots aired, impressions served, sessions held. Measures of effectiveness record whether the audience did what the objective named. Performance figures are large, easy to collect, and prove only that the plan was carried out. Report them, and never offer them as evidence that the campaign worked. For effectiveness, define impact indicators: specific, observable signs tied to the objective, read from the same record you used for the baseline in stage one.
Read the result against the baseline and the other causes
Set the result beside the baseline for the same period and the same source, then work through the list of other causes you wrote in stage one. If you held a channel back from a comparison area, or can compare with a similar place that had no campaign, use that. Claim contribution, with a range, and say what you cannot separate. A campaign that takes sole credit for a change that a new rule, a new service, or the season also drove is making an unsourced claim, and the audience for that claim is your funder.
Report null and negative results
The metrics were fixed before testing; use those and no others as the headline. If the objective was missed, say so in the first paragraph. If a channel showed no detectable effect, say that too, with the spend. Null results are the most useful thing a campaign can hand to the next one, and the least likely to be written down. Do not redefine success after the fact, switch to whichever indicator moved, or bury the shortfall under reach figures.
Measure what it cost the audience
The stage question has a second half: what did it cost the audience? Count complaints, requests to stop, and corrections. Ask a sample of the audience, with consent, whether they felt pressured, judged, or misled, and whether their trust in the messengers went up or down. Look beyond the audience to anyone the campaign may have stigmatised.
Then ask the question campaigns least like to ask: did we raise contempt for the people who disagree with us, or who did not act? Hostility between groups can be measured with a simple before-and-after survey item, and research on partisan animosity shows that communication can move it. A campaign that reaches its number by teaching one part of a community to despise another has done harm that its objective does not record.
Evaluate with the same restraint
Measure with the restraint you used in research. Prefer aggregate records. Do not track individuals across channels to prove an effect, and do not keep contact data past the date you set. Tell survey respondents who is asking and why. Hand the full results, favourable or not, to the partners and the approver before anyone drafts a press release.
Techniques at this stage, and where the line is
Use dashboards for what they measure: delivery and reach. Label those figures as performance.
The line: Presenting reach or engagement as proof of behaviour change misleads funders and the public about what was achieved.
Judge each decision by what was known when it was made, whatever the result.
The line: Citing a good outcome to excuse, after the fact, a tactic that should have failed the gate is how limits erode.
Compare the result with the predictions and thresholds you wrote down in advance.
The line: Rewriting the account so that the result looks as if it had been the plan all along deceives the next team.
Check other causes and comparison areas before attributing any change to the campaign.
The line: Claiming the campaign caused a change that you know had other drivers is a false claim about your own work.
There is no honest use. The headline is the metric fixed in advance, met or not.
The line: Redefining success once the data has arrived is the manipulative form in its entirety.
There is no honest use. Report every metric you set, including the harm indicators.
The line: A report assembled only from the favourable figures creates a false picture from true numbers.
Treat it as a harm indicator: measure whether attitudes toward people who disagree, or who did not act, got worse.
The line: Accepting or cultivating hostility toward another group because it lifts the numbers is a harm, whatever the objective.
Worked example
Fictional example, continued. The Riverside Health Partnership campaign ended on 22 August. The objective was 1,100 catch-up visits; the baseline for the same ten weeks last year was 610.
- 1Performance: 1,720 letters, 940 first text reminders, 120 radio spots, 38 evening and Saturday clinic sessions, and about 310,000 advertising impressions. The report lists these as delivery and nothing more.
- 2Effectiveness: the registry shows 1,024 catch-up visits, 414 more than the year before, an increase of 68 percent. The objective was missed by 76 visits. The number of children behind fell from 2,140 to 1,290.
- 3Other causes: the state sent its school-entry notices in July, and 612 of the 1,024 visits (60 percent) took place at the new evening and Saturday sessions. The report says the Partnership cannot divide the increase between easier access, the state notice, and its own messages, and claims a contribution only.
- 4Null result: during the five weeks when social advertising ran in two zip codes and not the third, visit rates showed no detectable difference. The report says so, gives the 9,000 dollar spend, and recommends not repeating it.
- 5Harm monitoring: nine complaints (six about the registry-based letter, three about text timing), 14 requests to stop, one correction. In an anonymous follow-up survey of 280 parents, 6 percent say they felt pressured, under the 10 percent threshold. Agreement with the statement that parents who do not vaccinate are bad parents is 22 percent, against 21 percent on the same item in the stage-two survey: no meaningful change.
- 6The full results go to the partners and to Dr Brandt before any announcement is drafted.
Outcome. A report whose first sentence reads: the campaign did not reach its objective of 1,100 visits; it recorded 1,024, against 610 the year before.
Evaluation record
Fill this in from the records, before anyone writes the summary. Saved in this browser only.
What was delivered: items, sessions, spend. These show the plan was carried out.
The objective, the baseline, and the result, all from the same record and period.
Each cause listed in stage one, and what you can and cannot say about its share.
Choose the strongest statement the evidence supports.
Channels or products that showed no effect or a bad one, with spend.
Tick each check you actually ran.
What the checks showed, including anything you would prefer not to report.
What evaluation data was collected, with what consent, and when it will be deleted.
Ethics gate: Phase VII, evaluation
0/7If any of these is not true: Do not publish or brief the results yet. Run the missing measure or reword the claim. If harm was found, take it to stage nine as the first item, and stop any activity that is still causing it.
Common mistakes
- ×Presenting reach as the result.
- ×Taking sole credit for a change that had several causes.
- ×Changing the headline metric after seeing the data.
- ×Reporting that no harm was found when no harm was looked for.
- ×Issuing the press release before the partners have seen the full results.
How much of this actually worked on people, and how would anyone know?
Related
References
- Department of the Army (2013). FM 3-53: Military Information Support OperationsEvaluation as the final phase, and the distinction between measures of performance and measures of effectiveness.
- Wakefield, M. A., Loken, B., & Hornik, R. C. (2010). Use of mass media campaigns to change health behaviour. The LancetCampaign effects depend heavily on concurrent services and policies, so attribution must account for other causes.
- Noar, S. M. (2006). A 10-year retrospective of research in health mass media campaigns: Where do we go from here?. Journal of Health CommunicationThe importance of outcome evaluation with sound designs, and the modest size of typical effects.
- Rice, R. E., & Atkin, C. K. (Eds.) (2013). Public Communication Campaigns (4th ed.). SageStandard practice for process, outcome, and impact evaluation of campaigns.
- Campbell, D. T. (1979). Assessing the impact of planned social change. Evaluation and Program PlanningIndicators used for decisions come under pressure to be distorted; a reason to fix metrics in advance and report all of them.
- Voelkel, J. G., Stagnaro, M. N., Chu, J. Y., et al. (2024). Megastudy testing 25 treatments to reduce antidemocratic attitudes and partisan animosity. ScienceAnimosity toward opponents is measurable and responds to communication, so it can be tracked as a campaign side effect.