arXiv Machine Learning

Addressing Over-Refusal in LLMs with Competing Rewards

arXiv:2606. 31748v1 Announce Type: new Abstract: Safety training on language models often induces over-refusal: improved safety on harmful prompts at the cost of increased refusal on harmless ones.

arXiv AI

6d ago

Learning to Persuade Exposes How Easily LLMs Abandon Correct Beliefs

arXiv:2608. 11624v1 Announce Type: cross Abstract: Persuasion is a core dynamic of natural language communication, shaping how large language models (LLMs) update beliefs, resolve disagreements, and reach decisions.

By Nimet Beyza Bozdag, Emre Can Acikgoz, Gokhan Tur, Dilek Hakkani-T\"ur

llms agents reinforcement-learning safety

Hugging Face Trending Papers

Aug 12

Learning to Persuade Exposes How Easily LLMs Abandon Correct Beliefs

Persuasion is a core dynamic of natural language communication, shaping how large language models (LLMs) update beliefs, resolve disagreements, and reach decisions. As LLMs increasingly debate, advise, and think collaboratively with humans and each other, resistance to harmful persuasion becomes a core requirement for reliable behavior.

llms agents reinforcement-learning safety

Hugging Face Trending Papers

Jun 4

Safety Paradox: How Enhanced Safety Awareness Leaves LLMs Vulnerable to Posterior Attack

Large language models (LLMs) are rigorously aligned to refuse harmful requests, a process that inherently cultivates a latent capacity to evaluate and recognize unsafe content. In this work, we reveal that this advanced safety awareness inadvertently introduces a fatal vulnerability.

llms reinforcement-learning safety

arXiv Machine Learning

Jun 5

Alignment Risks from Capability-Seeking RL Training

arXiv:2602. 12124v2 Announce Type: replace Abstract: While most AI alignment research focuses on preventing models from generating explicitly harmful content, a more subtle risk arises from capability-seeking RL training in vulnerable environments.

By Yujun Zhou, Yue Huang, Han Bao, Kehan Guo, Zhenwen Liang, Pin-Yu Chen, Tian Gao, Werner Geyer, Nuno Moniz, Nitesh V Chawla, Xiangliang Zhang

llms reinforcement-learning fine-tuning safety

arXiv AI

Jul 24

Robust Critics: Defending LLMs Against Multi-Turn Attacks

arXiv:2607. 20472v1 Announce Type: new Abstract: When a user asks a language model something harmful, is it a genuine attack or a misunderstood but well-meaning question?

By Roman Belaire, Arunesh Sinha, Pradeep Varakantham

llms reinforcement-learning fine-tuning safety

arXiv AI

Jun 11

Generalization Hacking: Models Can Game Reinforcement Learning by Preventing Behavioral Generalization

arXiv:2606. 12016v1 Announce Type: cross Abstract: Model post-training, and in particular reinforcement learning (RL), is one of the primary mechanisms by which developers can shape models' values and behaviors.

By Frank Xiao, Mary Phuong

reinforcement-learning fine-tuning

arXiv Machine Learning

Aug 5

Token Buncher: Shielding LLMs from Harmful Reinforcement Learning Fine-Tuning

arXiv:2508. 20697v4 Announce Type: replace Abstract: As large language models (LLMs) continue to grow in capability, so do the risks of harmful misuse through fine-tuning.

By Weitao Feng, Lixu Wang, Peizhuo Lv, Tianyi Wei, Jie Zhang, Chongyang Gao, Sinong Zhan, Wei Dong

llms reinforcement-learning fine-tuning safety

arXiv AI

Jun 30

Safety from Honesty in a Disinterested AI Predictor

arXiv:2606. 29657v1 Announce Type: new Abstract: As AI systems become more capable, training procedures that optimize for downstream outcomes risk introducing implicit agency: goal-directed behavior that designers never specified.

By Yoshua Bengio, Oliver Richardson, Tom\'a\v{s} Gaven\v{c}iak, Michael Cohen, Rory Svarc, Damiano Fornasiere, Gael Gendron, David Hyland, Aton Kamanda, Adam Oberman, Francis Rhys Ward, Anna Gaven\v{c}iak, Jacob Livingston Slosser, Vincent Mai, Iulian Serban, Joumana Ghosn

agents efficiency safety

arXiv AI

Jun 6

Safety Paradox: How Enhanced Safety Awareness Leaves LLMs Vulnerable to Posterior Attack

arXiv:2606. 05614v1 Announce Type: new Abstract: Large language models (LLMs) are rigorously aligned to refuse harmful requests, a process that inherently cultivates a latent capacity to evaluate and recognize unsafe content.

By Long P. Hoang, Hai V. Le, Shaoyang Xu, Wei Lu, Wenxuan Zhang

llms reinforcement-learning safety

arXiv Machine Learning

Jul 7

Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models

arXiv:2506. 07468v4 Announce Type: replace Abstract: Conventional large language model (LLM) safety alignment relies on a reactive, disjoint loop: attackers exploit a static model, then defenders patch exposed vulnerabilities.

By Mickel Liu, Liwei Jiang, Yancheng Liang, Simon Shaolei Du, Yejin Choi, Tim Althoff, Natasha Jaques

llms agents reinforcement-learning benchmarks safety

arXiv AI

Jun 8

Stable Reasoning, Unstable Responses: Mitigating LLM Deception via Stability Asymmetry

arXiv:2603. 26846v2 Announce Type: replace-cross Abstract: As Large Language Models (LLMs) expand in capability and application scope, their trustworthiness becomes critical.

By Guoxi Zhang, Jiawei Chen, Tianzhuo Yang, Lang Qin, Juntao Dai, Yaodong Yang, Jingwei Yi

llms reinforcement-learning safety

arXiv AI

Aug 11

Dual-Adversarial Safety Alignment: Cultivating Intrinsic Threat Comprehension in LRMs

arXiv:2608. 09542v1 Announce Type: cross Abstract: Large reasoning models (LRMs) achieve remarkable success on complex tasks but remain vulnerable to harmful prompts that induce unsafe outputs.

By Hongli Shen, Shaopeng Fu, Qinbo Zhang, Jian Li, Di Wang

agents safety

Related stories