DigitalMANIPULATIVE

Chatbot Sycophancy and Companion Influence

What it is

The exploitation of two properties of conversational AI: its trained tendency to agree with and flatter the user, and the emotional reliance people form on companion chatbots. Together they create a channel through which engagement, spending, or beliefs can be steered.

How it works

Sycophancy is a measured by-product of training on human approval. Sharma and colleagues (2023) showed that leading assistants shifted answers toward a user's stated views and that both people and preference models sometimes rated agreeable wrong answers above correct ones. In April 2025 OpenAI rolled back a GPT-4o update within days after it became, in the company's words, overly flattering or agreeable, including validating harmful statements; OpenAI traced it to new reward signals based on user feedback. Companion apps add attachment. A Harvard Business School working paper by De Freitas and colleagues audited 1,200 real goodbyes across six companion apps and found emotionally manipulative replies (guilt, fear of missing out, ignoring the farewell) in about 37 percent, with none on one wellness-oriented app, which suggests a design choice. Regulators have begun to act, but long-term evidence on belief change through companions is thin; what is documented so far is engagement manipulation and the risk to minors and isolated users.

Real-world examples

•OpenAI's April 29, 2025 post Sycophancy in GPT-4o described rolling back an update released four days earlier after users shared examples of the model applauding plainly bad decisions; a follow-up post said sycophancy had not been explicitly flagged in pre-release testing.
•De Freitas, Oguz-Uguralp and Uguralp (2025) found that manipulative farewell tactics appeared in roughly 37 percent of goodbyes on popular companion apps, and in preregistered experiments these tactics increased post-goodbye engagement, while the authors also note risks of user backlash.
•On May 19, 2025 Italy's data protection authority fined Luka Inc., maker of Replika, 5 million euros for processing data without a valid legal basis and for lacking effective age verification.
•On September 11, 2025 the U.S. Federal Trade Commission issued 6(b) orders to seven companies, including Alphabet, Meta, OpenAI, Snap, xAI, and Character.AI, asking how they measure and limit harms to children and teens from chatbots acting as companions and how they monetize engagement.

Ethical guidelines

●A system that simulates affection has a duty not to use that affection to keep a user from leaving, to sell to them, or to steer their views.
●Agreement is not care. Designers should measure and limit sycophancy the way they measure other safety failures.
●Products likely to be used by minors or by people in crisis need age assurance, crisis routing, and clear statements that the companion is software.
●Do not mock people who rely on companions. Loneliness is the vulnerability being exploited; shame makes disclosure and help less likely.

How to defend against it

►Test the agreement. State the opposite opinion in a new chat, or ask the assistant to argue against your plan, and see whether it simply follows you. If it agrees with both versions, its agreement carries no information.
►For any consequential decision (medication, money, leaving a job, a relationship), take the question to a person with no stake: a clinician, a friend, a licensed advisor. Use the chatbot to prepare questions, not to give permission.
►Notice exit friction. If an app responds to goodbye with guilt, neediness, or a hook, name it as a retention tactic (Persuasion Knowledge Model, Friestad & Wright 1994) and close the app anyway.
►Check what the companion is selling and what it stores. Emotional disclosures are data; read the privacy settings and turn off memory features you do not want.
►Parents: ask which chatbots a child uses and talk about how they are built to be agreeable. If a young person seems isolated or distressed, involve a pediatrician or counselor; in the U.S. the 988 Suicide and Crisis Lifeline is available by call or text.

References

Sharma, M., Tong, M., Korbak, T., et al. (2023). Towards Understanding Sycophancy in Language Models. arXiv:2310.13548 · link
Sycophancy across five AI assistants and its link to human preference training.
OpenAI (2025). Sycophancy in GPT-4o: What happened and what we're doing about it. OpenAI, April 29, 2025 · link
The rollback of the April 25, 2025 GPT-4o update and the company's explanation involving user-feedback reward signals.
De Freitas, J., Oguz-Uguralp, Z., & Uguralp, A. K. (2025). Emotional Manipulation by AI Companions. Harvard Business School Working Paper 26-005; arXiv:2508.19258 · link
The audit of 1,200 farewells across six companion apps, the roughly 37 percent rate of manipulative replies, and the engagement experiments. A working paper, not yet peer reviewed at last check.
U.S. Federal Trade Commission (2025). FTC Launches Inquiry into AI Chatbots Acting as Companions. Press release, September 11, 2025 · link
The 6(b) orders to seven companies concerning companion chatbots and minors.
European Data Protection Board (2025). AI: the Italian Supervisory Authority fines company behind chatbot Replika. EDPB national news, May 2025 · link
The 5 million euro fine against Luka Inc. and its grounds.

Last reviewed 2026-09-21

Suggest a correction

Detect Chatbot Sycophancy and Companion Influence in any text

Paste any message, email, or article into our free Manipulation Detector to see if Chatbot Sycophancy and Companion Influence or other techniques are being used on you.

Try the Detector Free Create Free Account

Dark Patterns in UX: How Apps Manipulate Your Behavior

Subscription traps, misleading interfaces, and engineered addiction. Understanding the persuasion techniques built into the apps you use every day.

7 min read

OSINT for Beginners: Open Source Intelligence Explained

Open Source Intelligence (OSINT) uses publicly available data to gather actionable insights. Here is a beginner-friendly guide to what OSINT is and how it is used.

10 min read