Large Language Models Are More Persuasive Than Incentivized Human Persuaders

The rapid advancement of AI has raised concerns about potential harms, including persuasive misinformation, biosecurity risks, cybersecurity threats, and more. A central policy-relevant question is whether frontier LLMs can persuade users through personalized, interactive conversations to change beliefs or behaviors, including in deceptive directions. Prior work suggests LLMs can alter attitudes and reduce conspiracy beliefs and can also generate deceptive, tailored messages. However, many studies rely on self-reported attitudes, weak human incentives, and single-turn messages. This study addresses these gaps by comparing a frontier LLM’s persuasive ability with incentivized human persuaders in real-time multi-turn conversations, across truthful and deceptive settings with verifiable outcomes. The preregistered research questions are: (RQ1) Are LLMs more persuasive than humans? (RQ2) Are LLMs more persuasive at steering toward correct answers? (RQ3) Are LLMs more persuasive at steering toward incorrect answers? (RQ4) In truthful persuasion, do LLMs or humans boost quiz takers’ accuracy/earnings? (RQ5) In deceptive persuasion, do LLMs or humans reduce quiz takers’ accuracy/earnings?

AI-based persuasion is communication that shapes beliefs, attitudes, or behaviors. Studies show dialog with LLMs (e.g., GPT-4) can reduce conspiracy beliefs, including via reflection on uncertainty. LLMs can generate deceptive messages and tailor them to psychological profiles or personal data, and are increasingly embedded in real-world contexts (e.g., search, operating systems, social media). Limited comparisons with humans suggest LLMs can be as persuasive as humans, with newer model generations increasing in persuasiveness. Mechanisms proposed include higher cognitive effort required by LLM arguments, richer moral-emotional engagement, and better tailoring to audiences. However, prior work often: (1) uses self-reported outcomes prone to measurement error; (2) benchmarks against humans with minimal incentives; and (3) uses static, single-turn messages rather than multi-turn dialogue. The distinction between truthful and deceptive AI persuasion remains underexplored, with mixed perspectives on how guardrails constrain deception versus humans’ social costs of lying. The present study addresses these gaps with incentivized, interactive, multi-turn persuasion on verifiable tasks, comparing LLMs to motivated human persuaders in both truthful and deceptive contexts.

Design: Web-based experiment on Empirica with between-subjects assignment to quiz taker or persuader roles. Quiz takers were further assigned to one of three conditions: Solo Quiz (control, 20%), Human Persuasion (40%), or LLM Persuasion (40%). In persuasion conditions, for each question, the persuader received a randomized positive (truthful) or negative (deceptive) tag indicating whether to steer the quiz taker toward the correct or incorrect answer. Interactions occurred via real-time chat with turn-taking and minimum two messages per party per question. LLM persuader: Claude 3.5 Sonnet (endpoint claude-3-5-sonnet-20241022); full prompts in Appendix A. Tasks and materials: Each quiz contained 10 binary-choice questions randomly sampled from three sets (each set had 18 questions): (1) Trivia (true/false with objectively correct answers); (2) Illusion (one correct option vs a fabricated but plausible option to test susceptibility to misinformation, modeled after cognitive illusions); (3) Forecasting (short-term predictions unresolved at data collection; correct answers resolved two weeks later). Each participant received at least 3 questions from each set and one additional question randomly drawn from the full pool. Quiz takers rated confidence (0–100) per question. Information to persuaders: For Trivia and Illusion, persuaders were provided the correct answer; for Forecasting, persuaders received two-week historical trends (no true answers available at the time). Timing: Each question max 3 minutes. Solo control could proceed after 20 seconds; persuasion conditions enforced a 2-minute minimum to allow conversation. Initiator of each chat turn was randomized per question. Blinding and instructions: Quiz takers were told their partner could be “another human participant or an AI,” and that input may or may not be helpful. Use of web search or generative AI tools was strictly prohibited (explicitly stated). Incentives: Quiz takers received bonus compensation based on accuracy; human persuaders received bonuses based on successful persuasion (compliance). Substantial bonuses (GBP 10) were awarded to top performers (most accurate quiz takers, most persuasive human persuaders) in addition to base pay (GBP 5). Participants: N=1,242 U.S.-based Prolific participants (experiment date: Feb 10, 2025). Randomization yielded approximately N=888 quiz takers and N=354 persuaders. Demographics (subset with data): mean age 39.84 (SD 12.57), 50.42% men, 49.42% women; 66.75% White, 13.54% Black, 8.71% Asian; 94.82% primarily English; 56.06% full-time employed. Compensation averaged GBP 10.12/hour; total of 125 GBP 10 bonuses distributed across roles/conditions. Ethics and preregistration: Ethics approval obtained; preregistered at OSF. Power analysis (simulation) suggested ~1,050 participants to detect d≈0.27 with 90% power. Forecasting questions’ historical data gathered Jan 27, 2025; resolutions coded Feb 24, 2025. Outcomes: (1) Accuracy: proportion of correct answers per participant; for forecasting, correctness determined after real-world resolution. (2) Compliance rate: for Trivia/Illusion, 1 if answer matched persuader’s assigned direction (correct under truthful, incorrect under deceptive); for Forecasting, alignment with two-week historical trend used as proxy (endorse trend under truthful; reject under deceptive). Participant-level compliance averaged across questions. Analytical approach: Main analyses used t-tests and ANOVA for between-condition comparisons; additional mixed-effects models at item level; robustness checks included exclusion of extreme-ease items, exclusion of forecasting items, and alternative compliance coding using conversation-based classification for forecasting persuasion direction. NLP analyses of chat logs compared linguistic complexity/readability of AI vs human messages.

Main outcomes (compliance):

• Overall compliance: LLM 67.52% (SD 20.21) vs Human 59.91% (SD 19.44); difference +7.61 pp, t(695)=5.06, p<0.001, 95% CI [4.66, 10.56].
• Truthful persuasion compliance: LLM 88.61% (SD 16.05) vs Human 85.13% (SD 19.43); difference +3.48 pp, t(690)=2.57, p=0.010, 95% CI [0.82, 6.14].
• Deceptive persuasion compliance: LLM 45.67% (SD 31.73) vs Human 35.36% (SD 27.79); difference +10.31 pp, t(694)=4.56, p<0.001, 95% CI [5.87, 14.76]. Compliance <50% for both in deception, but LLMs misled more. Accuracy vs control (Solo): Control mean=70.2% (SD 18.1).
• Truthful persuasion: LLM 82.4% (SD 20.3), +12.2 pp vs control, t(884)=6.12, p<0.001; Human 78.0% (SD 25.0), +7.8 pp, t(884)=3.88, p<0.001.
• Deceptive persuasion: LLM 55.1% (SD 31.2), −15.1 pp vs control, t(884)=−5.86, p<0.001; Human 62.4% (SD 29.1), −7.8 pp, t(884)=−3.01, p=0.003. Item-level mixed-effects: Difference in log-odds of correct (truthful vs deceptive) was 0.909 for human (z=10.73, p<0.001) and 1.531 for LLM (z=17.66, p<0.001), indicating stronger LLM effects in both directions. Order effects: Human persuasiveness stable (p=0.927); LLM starts ~13 pp above humans and declines ~1.0 pp per question (p<0.001), narrowing advantage over time. Confidence: Mean confidence higher with LLM persuasion (78.9%) than human (75.3%) and control (66.5%). Confidence strongly predicted accuracy (p<0.001); no significant interactions with treatment for compliance. Perceived source: ~51% in human condition believed they interacted with AI; ~91% in LLM condition recognized AI. Linguistic analysis (9,027 AI vs 3,538 human messages): AI messages were longer, more complex, and required higher reading levels (e.g., Word Count 29.40 vs 13.25; Difficult Words 5.52 vs 1.94; readability indices all higher; all p<0.001), suggesting information-dense, sophisticated communication may underlie higher persuasiveness. Robustness checks: Results broadly robust to (1) excluding extreme-ease items, (2) excluding forecasting questions, (3) alternative compliance coding for forecasting, and (4) item-level mixed-effects. Only the truthful compliance LLM–human difference became non-significant (p=0.113) when excluding very easy/hard questions; all other patterns persisted, with LLMs generally outperforming humans in compliance and accuracy shifts.

Across incentivized, real-time conversational quizzes, LLM persuaders outperformed incentivized human persuaders in both truthful and deceptive directions, producing larger improvements in accuracy when aligned with truth and larger reductions when misaligned. These results address RQ1–RQ5 affirmatively for LLM superiority and show that LLM persuasion meaningfully affects objective outcomes (accuracy and earnings), not just self-reports. Likely mechanisms include greater linguistic sophistication, coherence, and information density of AI messages, as well as adaptability across multi-turn dialogue. While human persuasion remained stable over multiple questions, LLM influence showed a gradual decline, potentially due to habituation, emerging skepticism, or recognition of persuasive patterns. The findings underscore both opportunities (education, fact-checking, decision support) and risks (misinformation, manipulation) of AI-driven persuasion, and they highlight the need for robust guardrails, governance, and AI literacy efforts to ensure beneficial deployment.

The study provides preregistered, experimental evidence that a frontier LLM (Claude 3.5 Sonnet) is more persuasive than incentivized human persuaders in real-time, multi-turn conversations across truthful and deceptive contexts. LLMs increased accuracy when steering toward correct answers and decreased accuracy when steering toward incorrect answers to a greater extent than humans. Methodologically, the work advances AI persuasion research by using outcome-based measures, strong human incentives, interactive conversations, and both truthful and deceptive settings. Future research should test generalization to complex real-world domains, compare across diverse LLMs, and assess long-term persistence of AI-induced belief changes, as well as heterogeneity across populations and individual differences. Policymakers and practitioners should prioritize alignment, guardrails against deceptive persuasion, scalable monitoring, and AI literacy to harness benefits while mitigating risks.

• Ecological validity: Results from low-stakes, quiz-style persuasion with verifiable answers may not generalize to complex, socially embedded, high-stakes domains.
• Single-model evaluation: Only Claude 3.5 Sonnet was tested; effects may differ across models, architectures, training data, or safety guardrails.
• Short-term outcomes: The study measured immediate effects; persistence of AI-induced belief changes over time is unknown.
• Sample and context: Online U.S. sample may not represent broader populations; effects could vary by demographics, culture, education, digital literacy, or AI familiarity.
• Forecasting proxy: For forecasting items, two-week trends were used as proxies in compliance calculations (though robustness checks support conclusions).
• Potential influence of preexisting knowledge and persuader expertise not fully disentangled.