arXiv AI
Understanding and Reducing Misalignment in Persuasion Optimization
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
Sep 1
Reinforcement Learning Can Amplify Emergent Misalignment from Harmless Rewards
arXiv:2605.31328v2 Announce Type: replace Abstract: Emergent misalignment (EM) is the surprising tendency of language models to become broadly misaligned after fine-tuning on narrowly misaligned exam...
arXiv AI
Oct 1
Alignment via Training Against Probes Without Losing Monitorability
The paper proposes probe-guided fine-tuning, a method that uses probes detecting undesired properties in model activations as a direct training signal. Experiments show that continuously updated probes reduce harmfulness and improve honesty while preserving utility, outperforming DPO and inference-time steering in safety-utility trade-offs and robustness to jailbreak and abliteration attacks. Importantly, the concepts remain linearly encoded after fine-tuning, maintaining monitorability.
arXiv AI
Jun 17
Using Cognitive Models to Improve Language Model Simulation of Human Persuasion Games
arXiv:2606. 17657v1 Announce Type: new Abstract: People make decisions differently in strategic interactions.
arXiv AI
Oct 7
Rationale-Guided Policy Optimization: Learning to Reason with Adaptive Rationale Scaffolding
Rationale-Guided Policy Optimization (RGPO) is a reinforcement‑learning framework that adaptively uses ground‑truth rationale information to scaffold a language model’s reasoning process. Instead of treating reference solutions as fixed imitation targets, RGPO temporarily incorporates rationales to help the model generate better responses, then reverts to unguided learning with higher‑reward, model‑generated solutions. Experiments in both language‑only and vision‑language tasks show that RGPO consistently outperforms RLVR baselines, with ablation studies confirming that adaptive rationale guidance is a key factor in its success.
arXiv AI
Aug 13
Learning to Persuade Exposes How Easily LLMs Abandon Correct Beliefs
arXiv:2608. 11624v1 Announce Type: cross Abstract: Persuasion is a core dynamic of natural language communication, shaping how large language models (LLMs) update beliefs, resolve disagreements, and reach decisions.
arXiv AI
Aug 18
The Fragility of Strategic Thinking in Large Language Models
arXiv:2510. 10813v2 Announce Type: replace Abstract: Large Language Models (LLMs) are increasingly applied to domains that require reasoning about other agents' behavior, such as negotiation, policy design, and market simulation.