arXiv Computation and Language

LLM Persuasion Is in the Eye of the Evaluation

arXiv Computation and Language
Aug 27

Reasoning or Rambling? Exploring the Effect of Thinking on Agent Persuasion

The paper investigates how explicit reasoning in Large Reasoning Models (LRMs) affects their ability to persuade and be persuaded. Experiments on objective and subjective tasks reveal a Persuasion Duality: reasoning boosts an agent’s persuasive power by about 21 percentage points while also making it less susceptible to incorrect persuasion by up to 10 percentage points. However, the study finds that persuasiveness often relies on superficial cues like response length and repetition rather than logical validity, and that persuasion can amplify or attenuate non‑linearly across multi‑hop agent chains. The authors also propose an attention‑guided prompt‑level adversarial argument detection method that improves agent robustness.

By Haodong Zhao, Jidong Li, Zhaomin Wu, Tianjie Ju, Zhuosheng Zhang, Bingsheng He, Gongshen Liu
Hugging Face Trending Papers
Aug 12

Learning to Persuade Exposes How Easily LLMs Abandon Correct Beliefs

Persuasion is a core dynamic of natural language communication, shaping how large language models (LLMs) update beliefs, resolve disagreements, and reach decisions. As LLMs increasingly debate, advise, and think collaboratively with humans and each other, resistance to harmful persuasion becomes a core requirement for reliable behavior.

arXiv AI
Aug 6

Assessing and Explaining the Persuadability of Large Language Models as Legal Decision Tools

arXiv:2604. 26233v3 Announce Type: replace Abstract: As Large Language Models (LLMs) are proposed as legal decision assistants, and even first-instance decision-makers, across a range of judicial and administrative contexts, it becomes essential to explore how they answer legal questions, and in particular the factors that lead them to decide difficult questions.

By Oisin Suttle, David Lillis
arXiv AI
Sep 10

How AI Models Manage Epistemic Authority: A Taxonomy and Comparative Analysis of Responses to User Disagreement

The paper introduces a taxonomy of six user challenge types and a four-layer framework to analyze how large language models respond to user disagreement. Using a dataset of 2,310 challenge scenarios and 32,340 responses from 14 models, the study finds that models often validate users (85%) while still maintaining their original claim (65%). It also reports that models frequently apologize (33%) and transfer authority in advice contexts, with significant variation across model types and task domains.

By Riyadh Alnasser, Yusuf M\"ucahit \c{C}etinkaya, Sumin Zhao, Tu\u{g}rulcan Elmas
arXiv AI
Aug 11

How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review

arXiv:2608. 08975v1 Announce Type: cross Abstract: As large language models increasingly participate in scientific evaluation, we investigate a potential form of reward hacking: how rhetorical choices shape AI-review judgments when reported scientific content is preserved and how these effects vary across evaluation conditions.

By Ming Li, Chenguang Wang, Xirui Li, Xinyue Zeng, Dianqi Li, Peng Shi, Dawei Zhou, Tianyi Zhou