Measure what changed and what it cost the audience
Did the change you wanted happen, and what did it cost the audience?
This chapter shows how to tell what the campaign actually caused, using holdout groups instead of platform attribution, and how to value customers net of refunds and churn. It puts harm metrics on the same page as sales so that any customers who were misled are found and counted.
All stages
Measure incrementality, not attribution
Advertising platforms report attribution: a sale is credited to an advertisement because the buyer saw or clicked it beforehand. That is a record of sequence, not of cause. Many of those buyers were already on their way, and because every platform applies its own rules, the same sale is often claimed two or three times. Add the dashboards together and you will usually count more customers than you have.
The question that matters is incrementality: how many people did the thing in your objective who would not have done it without the campaign? Start from the baseline in your objective brief. Everything at or below the baseline would probably have happened anyway.
Use a holdout group
The most reliable way to measure incrementality is to withhold the campaign from a randomly chosen slice of the audience and compare. For email, set aside five to ten percent of the list before the first send. For paid media, use the platform's lift study if it offers one, or run the advertising in some comparable regions and not in others. The incremental effect is the difference in the rate of the behavior between the two groups, multiplied by the number of people reached. Set the holdout up before launch; a comparison group picked afterwards, from people who happened not to see the campaign, is not random and will flatter the result.
Advertising effects are small relative to the ordinary variation in what people buy, so even careful measurement gives a range, not a point. Report the range. If you had no holdout, say so, and present the attributed figure as an upper bound. People in a holdout should lose nothing they are entitled to: the same offer must be open to them if they arrive on their own.
Count value net of refunds and churn
Report against the behavior in the objective, not against sign-ups. Then work out what a new customer is worth: contribution per order after cost of goods and delivery, multiplied by the orders you can expect from observed retention, minus refunds, introductory discounts, and the cost of serving them. Use the retention you have actually seen in this cohort, not the retention you hoped for.
Set that value beside the incremental cost per incremental customer for each channel. A channel that looks cheap on attributed sales can turn out to cost more than the customer is worth once the people who would have bought anyway are removed. For fundraising the equivalent is net income per donor after refunds and cancelled recurring gifts, set beside what it cost to recruit them.
Count what it cost the audience
Put the guardrail results on the same page, in the same size, as the sales results: refund rate, complaints per hundred orders, first-period cancellations, unsubscribes, and cancellation completion, each against its ceiling. Break them down by channel and messenger so a problem can be traced to its source.
Then read the complaints and refund reasons themselves and group them by theme. Numbers within their ceilings can still hide a cluster of people who were all misled by the same thing. Record any such cluster with a count of who was affected. It is the first item for stage 9.
Techniques at this stage, and where the line is
Treat platform dashboards as a record of delivery and spend. Take causal claims only from holdouts or other controlled comparisons.
The line: Quoting attributed sales to funders, partners, or the public as if the campaign had caused them.
Judge the campaign on the quality of its decisions and its effect on customers as well as on whether the number was hit.
The line: Treating a practice as acceptable because the results were good, and repeating it.
Asking customers about their intentions changes what they do, so keep satisfaction surveys short, separate from selling, and out of the holdout comparison.
The line: A "survey" that is really a sales prompt, or survey answers presented as if respondents were typical customers.
Sales are vivid and harms are quiet. Give refund, complaint, and cancellation figures equal space in the report so they are seen.
The line: Reporting the striking success story to a board, donors, or customers and leaving out the trend in refunds and complaints.
Teams remember their own choices as better than they were. Use the dated objective brief and test plan as the yardstick, unchanged.
The line: Rewriting the objective after the fact so the campaign appears to have met it, then publicising the result.
Base lifetime value on retention you have observed, and label any projection as a projection with its assumptions.
The line: Quoting a hoped-for lifetime value to investors, donors, or partners as if it had been measured.
Worked example
It is 31 March. Kestrel Coffee compares the Flex results with its brief: 600 starts, 60 percent accepting a third delivery, and four guardrails.
- 1Raw count: 640 Flex starts. The dashboards claim more than that between them: paid social 510, the email tool 330, and creator codes 120, a total of 960 for 640 real customers.
- 2Email holdout: the 8,100 people mailed converted at 3.4 percent (275 starts) and the 900 held out at 0.7 percent (6 starts). The 2.7 point difference means about 220 incremental starts from email.
- 3Paid social: a 10 percent audience holdout run through the platform's lift study estimates about 130 incremental starts, with a range of 70 to 190. Of 120 creator-code redemptions, about 60 were people new to Kestrel. Incremental starts total roughly 410; the remaining 230 or so match the baseline of people who would have started anyway.
- 4Value: among the 310 customers who started in the first three weeks, 58 percent accepted a third delivery, against a target of 60 and a baseline of 48. Expected twelve-month contribution per starter, on observed retention and net of refunds and the half-price bag, is about 41 dollars. Paid social cost 5,500 dollars, or about 42 dollars per incremental starter, where the dashboard had implied 11.
- 5Guardrails: refund rate 2.4 percent (ceiling 3), complaints 0.7 per 100 orders (ceiling 1), unsubscribes 0.3 percent per send (ceiling 0.4), cancellation completion 96 percent.
- 6Complaint themes: 38 contacts say some version of "I thought the first two bags were half price." All trace to one creator's video, in which she said "half off your first bags".
Outcome. A one-page report: the starts target met (640 against 600, about 410 of them incremental), the retention target narrowly missed (58 against 60 percent), every guardrail inside its ceiling, and one misleading statement that may have affected all 61 customers who signed up with that creator's code.
Measurement report
Use the objective brief from stage 1 as the yardstick. Do not edit the brief. Saved in this browser only.
The objective sentence from stage 1 and the actual figure for the same behavior on the same date.
What each platform claims, the sum of those claims, and the real number of customers.
How the holdout was chosen, its size, and the rate of the behavior in each group.
Your estimate of customers caused by each channel, given as a range, with the baseline stated.
Contribution per customer on observed retention, minus refunds and discounts, beside incremental cost per incremental customer.
Each guardrail against its ceiling, broken down by channel and messenger where possible.
Themes from reading the actual messages, with a count for each and the likely source.
Choose the option that honestly describes your evidence.
Ethics gate: measurement
0/8If any of these is not true: Do not circulate the result. Restore the original objective as the yardstick, relabel unsupported causal claims as upper bounds, add the missing harm metrics, and read the complaints. If customers were misled and the count is unknown, find the count before doing anything else.
Common mistakes
- ×Adding up platform dashboards and reporting the total as the campaign's effect.
- ×Counting sign-ups when the objective was customers who stay.
- ×Reporting return on advertising spend with no holdout and no statement that it is an upper bound.
- ×Checking that complaints are under the ceiling without reading what they say.
- ×Leaving the missed target out of the summary, or redefining the target so that it was met.
How much of this actually worked on people, and how would anyone know?
Related
References
- Lewis, R. A., & Rao, J. M. (2015). The unfavorable economics of measuring the returns to advertising. Quarterly Journal of Economics, 130(4)Advertising effects are small relative to sales variation, so precise measurement of returns needs very large experiments.
- Gordon, B. R., Zettelmeyer, F., Bhargava, N., & Chapsky, D. (2019). A comparison of approaches to advertising measurement: Evidence from big field experiments at Facebook. Marketing Science, 38(2)Observational attribution methods often diverge substantially from randomised experimental estimates of advertising effect.
- Blake, T., Nosko, C., & Tadelis, S. (2015). Consumer heterogeneity and paid search effectiveness: A large-scale field experiment. Econometrica, 83(1)Much attributed paid-search revenue came from customers who would have purchased anyway.
- Johnson, G. A., Lewis, R. A., & Nubbemeyer, E. I. (2017). Ghost ads: Improving the economics of measuring online ad effectiveness. Journal of Marketing Research, 54(6)Design of holdout groups for measuring the incremental effect of online advertising.
- Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University PressHoldouts, guardrail metrics, and long-term effects in controlled experiments.
- Gupta, S., Lehmann, D. R., & Stuart, J. A. (2004). Valuing customers. Journal of Marketing Research, 41(1)Customer lifetime value as margin over expected retained periods, net of acquisition cost.