Measurement: did it actually work on anyone?
How much of this actually worked on people, and how would anyone know?
This chapter separates what an operation produced from who saw it and from what, if anything, changed as a result. It summarises the evidence that most exposed operations had little measurable effect, introduces the Breakout Scale as a way to keep proportion, and explains why overstating an operation hands it a result it did not earn.
All stages
Output, reach and effect
Three different things get reported as the impact of an operation. Output is what was produced: accounts, posts, articles, adverts. Reach is who saw it. Effect is what changed in what people believe or do. Most public figures are output or reach, and neither is persuasion. A post seen a million times may have convinced nobody, may have been seen mostly by people who already agreed, or may have been viewed largely by the network's own accounts.
Establishing effect needs a comparison: what the same people believed before, or what similar people who were not exposed believe. That evidence is hard to get, and honest assessments say so. The January 2017 US intelligence assessment attributed activity to Russia and stated explicitly that it made no assessment of the impact on the election outcome.
What the research found
Where effect has been measured, it has been small. Eady and colleagues linked survey responses to the Twitter feeds of the same people during the 2016 US election. Exposure to Internet Research Agency accounts was heavily concentrated among a small share of users, most of them already strongly Republican, and the study found no evidence of a meaningful relationship between exposure and changes in attitudes, polarisation or voting behaviour. Bail and colleagues reached a similar conclusion for late 2017: the people who interacted with the accounts were already highly polarised, and no change in their attitudes was detected.
These studies cover one platform and one operation, and their authors say so. They do not measure other platforms, closed groups, or the hack-and-leak activity of the same year. They do fit the wider record. Secondary Infektion ran for six years with almost no engagement. OpenAI rated none of the operations it disrupted in 2024 above the second of six categories of spread.
The Breakout Scale
Ben Nimmo's Breakout Scale gives a common way to describe how far an operation travelled. Category One stays on one platform within one community. Category Two reaches several communities on one platform, or several platforms without leaving its starting community. Category Three breaks out on multiple platforms. Category Four crosses into mainstream media. Category Five is amplified by celebrities or other prominent figures. Category Six prompts a policy response or a call for violence.
The scale measures spread, not persuasion, and its value is proportion. Most exposed operations sit in the lowest categories. The jump that matters most is into mainstream coverage, and that step is taken by journalists and prominent users, who can decline to take it.
Perception hacking, and the opposite error
Meta uses the term perception hacking for efforts to make an operation seem larger and more effective than it is, so that the public concludes that elections, news and neighbours cannot be trusted. Overstating an operation therefore completes its work. Analysts who read leaked files from the company behind Doppelganger argue that its operators counted alarmed Western coverage as a result.
The opposite error is to conclude that nothing matters. Reach figures miss harms that fall on individuals: the dissident or journalist buried in hostile comments, the official who receives abuse after a doctored clip, the resident who stops speaking in public. Authentic leaked material has had real consequences where forgeries did not. A proportionate account states what was measured, by whom, on which platform, and what could not be seen.
Techniques you will meet here, and the tell
The operation, or someone describing it, inflates its apparent scale so that distrust becomes the effect.
The tell: The claimed impact rests on totals of posts and views, with no evidence about real audiences or changed minds.
An unwelcome result, protest or opinion is explained away as the product of an operation.
The tell: The existence of an operation is offered as proof that it caused the outcome, with nothing connecting the two.
A live tracker or chart of account activity is read as a measure of influence on the public.
The tell: The number on the dashboard is quoted without its inclusion rule, its error rate or its caveats.
People assume that propaganda works on others far more than on themselves, which inflates estimates of effect.
The tell: The speaker is sure the public was fooled and equally sure that they personally were not.
Only what is countable (views, shares, account totals) is treated as real, and harder questions about belief and harm are left out.
The tell: Every figure in the assessment is a count of activity and none is a measure of what changed.
Repetition can make a claim feel familiar, and familiar claims feel truer, without anyone being argued into anything. This is one route by which reach could matter.
The tell: People say they have heard the claim somewhere and cannot say where or on what evidence.
An effect that reach figures miss: people who believe they are outnumbered stop speaking, whatever they privately think.
The tell: Real participants have gone quiet in a space where the loudest accounts are the newest ones.
Worked example
The Alder Bay council has voted seven to four to approve the footbridge, with a condition on loading bays. A regional news site wants to run a piece headlined along the lines of fake news network targets Alder Bay, and asks Priya how much damage was done. She sets out what she can actually measure.
- 1She makes three columns. Output: about forty articles on the look-alike site and a few hundred posts. Reach: public counts on those posts. Effect: left empty until she has evidence.
- 2She looks at where the engagement came from. Most reactions on the coordinated posts came from the same cluster of accounts. Two items were shared widely by real residents: the cut clip of Councillor Okafor and the cost claim.
- 3She places the activity on the Breakout Scale. It began in two groups on one platform; the cost claim crossed to a regional forum and prompted one radio enquiry that did not air. She puts it at Category Two to Three, with a near miss at Four.
- 4She fills the effect column with what she can count. Of thirty-one letters to the Courier about the bridge, nine repeated the 4.1 million figure as the build cost. Three speakers at the public meeting cited it. So some real residents believed it. She has no evidence that the incomers theme changed anyone's view, and the vote matched what councillors had indicated beforehand. She cannot see private messaging groups and writes that down.
- 5She records the harm the counts miss. Councillor Okafor received abusive messages after the clip circulated, and a Harbourside trader told her he has stopped posting in the group.
- 6She tells the regional site what the evidence supports: a site imitating the Courier circulated a misleading cost claim that some residents repeated, and there is no evidence that it changed the vote. She asks them not to describe the town as having been swayed.
Outcome. Priya has an assessment that neither inflates nor dismisses. One claim reached and misled some real residents and is worth correcting. Most of the volume reached mainly itself. Two people were harmed in ways no view count shows. The larger story the regional site wanted would have given the campaign an importance it had not achieved.
Effect assessment
Fill this in about activity you have actually examined. Keep output, reach and effect in separate boxes and note where each number came from. Saved in this browser only.
Describe the posts, articles, accounts or adverts you observed, with approximate counts.
Record the figures you have, who published them, and how much of the engagement came from the network itself.
Choose the highest category the evidence supports.
List any evidence that real people changed a belief or an action, and what you are comparing it with.
Write down what else could account for it: people already held the view, events moved opinion, an open campaign made the case.
Note any harassment, intimidation or withdrawal from public discussion affecting specific people.
Name the places and measures you had no access to, such as private groups or other platforms.
Write one sentence on scale and effect that you could defend line by line.
Decision check: how much of this worked?
0/7If any of these is not true: Report only what was observed and say plainly that the effect is unknown. Do not publish or repeat totals without their source and their limits. If you are being asked for a dramatic figure, give the proportionate sentence instead. If individuals have been harmed, deal with that directly, whatever the overall scale.
Common mistakes
- ×Counting impressions as minds changed. Reach is delivery, and much of it is the network looking at itself.
- ×Doing the operation's publicity for it. A headline about a vast campaign with meagre real engagement is the outcome a perception-hacking effort wants.
- ×Over-attribution of outcomes: explaining a result you dislike by pointing to an operation, when nothing connects the two.
- ×Swinging to dismissal. Small measured effects on opinion do not cancel harassment of individuals, the cost to trust, or the damage a genuine leak can do.
- ×Generalising from one study. Findings about one platform in one year say little about closed messaging, other countries or the next operation.
Did the change you wanted happen, and what did it cost the audience?
Related
References
- Eady, G., Paskhalis, T., Zilinsky, J., Bonneau, R., Nagler, J., & Tucker, J. A. (2023). Exposure to the Russian Internet Research Agency foreign influence campaign on Twitter in the 2016 US election and its relationship to attitudes and voting behavior. Nature CommunicationsExposure was concentrated among a small group of already partisan users, with no evidence of a meaningful relationship to attitudes or voting.
- Nimmo, B. (2020). The Breakout Scale: Measuring the Impact of Influence Operations. Brookings InstitutionThe six categories used to describe how far an operation spread.
- Bail, C. A., Guay, B., Maloney, E., Combs, A., Hillygus, D. S., Merhout, F., Freelon, D., & Volfovsky, A. (2020). Assessing the Russian Internet Research Agency's impact on the political attitudes and behaviors of American Twitter users in late 2017. Proceedings of the National Academy of SciencesNo detectable effect of interaction with the accounts on attitudes; those who interacted were already highly polarised.
- Meta (2021). Threat Report: The State of Influence Operations 2017-2020Perception hacking: operations that seek to appear more effective than they are.
- OpenAI (2024). AI and Covert Influence Operations: Latest TrendsBreakout Scale ratings for disrupted operations, none above Category Two.
- Graphika (Nimmo, B., Francois, C., Eib, C. S., Ronzaud, L., et al.) (2020). Exposing Secondary Infektion. GraphikaA six-year operation that drew almost no engagement, with one exception involving apparently authentic documents.