DigitalMANIPULATIVE

Synthetic Personas at Scale

What it is

The use of generative models to produce the names, biographies, posting histories, and voices of fake accounts in bulk, so that one operator can present as a crowd of distinct, fluent individuals.

How it works

A persona network used to be limited by labor: each fake account needed a person to write in character, and non-native operators gave themselves away through language errors and recycled text. Language models remove much of that cost. Enforcement reports since 2024 describe operators generating account names and bios, short comments in many languages, and replies to their own posts to simulate a conversation. What the reports also show is the limit: fluent text does not supply an audience. OpenAI's May 2024 report rated none of five disrupted operations above Category 2 on the Brookings Breakout Scale, and Meta reported that generative AI had not impeded its ability to detect networks, because detection rests on behavior (account creation patterns, shared infrastructure, synchronized activity) rather than on prose quality. The lever being pulled is the consensus heuristic; the persona is only the costume.

Real-world examples

  • •In May 2024 OpenAI reported that the Russia-origin operation it called Bad Grammar and the Israeli commercial firm STOIC (nicknamed Zero Zeno) used its models to generate large volumes of short comments posted on Telegram, X, and Instagram, and that STOIC generated replies to its own posts to fake engagement. OpenAI assessed that none of this attracted authentic audiences.
  • •On July 9, 2024 the U.S. Justice Department announced the seizure of two domains and the search of 968 X accounts tied to a bot farm it attributed to an RT employee and Russian intelligence, run with software called Meliorator that generated fictitious personas, some posing as Americans, to post pro-Kremlin narratives.
  • •Meta's Q1 2024 Adversarial Threat Report described an Israel-based network whose fake accounts posed as Jewish students, African Americans, and concerned citizens in the U.S. and Canada, posting likely AI-generated comments that real users often answered by calling them propaganda.
  • •OpenAI's June 2025 threat report described a China-origin operation it named Sneer Review bulk-generating posts and replies, and an operation it named High Five generating pro-government comments in the Philippines, again with little evidence of authentic engagement.

Ethical guidelines

  • ●There is no honest deployment of invented citizens. Disclosed brand characters and labeled bots are a different thing; the deception here is about who is speaking and how many of them exist.
  • ●Persona operations counterfeit specific communities. The real students, veterans, or minority groups being impersonated bear the reputational cost.
  • ●Commercial firms that sell this service to governments or campaigns are operators, not vendors of neutral tools, and the enforcement record treats them that way.
  • ●Accusing real people of being AI personas without behavioral evidence is its own harm; organic movements are routinely dismissed as bots.

How to defend against it

  • ►Stop grading the prose. Fluent, polite, grammatical text is no longer evidence of a real person. Look at behavior instead: account age, what else the account has ever posted, whether it only ever amplifies the same few accounts, and whether several accounts post near-identical points within minutes.
  • ►Check whether the account has a life outside the topic. Real people have local references, old posts, and friends who reply about other things.
  • ►Read laterally before treating a wave of agreement as a groundswell (Wineburg & McGrew): search for the claim and the outlet by name and see who independent of the thread is reporting it.
  • ►Treat reply counts and comment volume as unverified. Ask what the base rate of real opinion is from a source that samples people, such as a reputable poll, rather than counting posts.
  • ►Be honest about the limit: an individual reader often cannot tell. Platform and lab takedown reports (Meta, OpenAI, Microsoft, Graphika, DFRLab) are the reliable record of which networks were fake.

References

  1. OpenAI (2024). AI and Covert Influence Operations: Latest Trends. OpenAI threat intelligence report, May 30, 2024 · link
    Bad Grammar and Zero Zeno generating bulk comments and fake replies; no operation scored above 2 on the Breakout Scale; no meaningful increase in audience engagement.
  2. U.S. Department of Justice, Office of Public Affairs (2024). Justice Department Leads Efforts Among Federal, International, and Private Sector Partners to Disrupt Covert Russian Government-Operated Social Media Bot Farm. Press release, July 9, 2024 · link
    The Meliorator bot farm: two domains seized, 968 X accounts searched, AI-generated personas posing as Americans.
  3. Meta (2024). Adversarial Threat Report, First Quarter 2024. Meta Transparency Center, May 2024 · link
    The Israel-based network linked to STOIC, its impersonated identities, likely AI-generated comments, and Meta's finding that generative AI had not impeded detection.
  4. Goldstein, J. A., Sastry, G., Musser, M., DiResta, R., Gentzel, M., & Sedova, K. (2023). Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations. arXiv:2301.04246 (Georgetown CSET, OpenAI, Stanford Internet Observatory) · link
    The analysis that language models lower the labor cost of persona operations and the framework of mitigations.
Last reviewed
Suggest a correction

Detect Synthetic Personas at Scale in any text

Paste any message, email, or article into our free Manipulation Detector to see if Synthetic Personas at Scale or other techniques are being used on you.

Try the Detector FreeCreate Free Account

Related Articles

Dark Patterns in UX: How Apps Manipulate Your Behavior

Subscription traps, misleading interfaces, and engineered addiction. Understanding the persuasion techniques built into the apps you use every day.

7 min read

OSINT for Beginners: Open Source Intelligence Explained

Open Source Intelligence (OSINT) uses publicly available data to gather actionable insights. Here is a beginner-friendly guide to what OSINT is and how it is used.

10 min read