arXiv Machine Learning June 5, 2026 By Yujun Zhou, Yue Huang, Han Bao, Kehan Guo, Zhenwen Liang, Pin-Yu Chen, Tian Gao, Werner Geyer, Nuno Moniz, Nitesh V Chawla, Xiangliang Zhang

Alignment Risks from Capability-Seeking RL Training

Read the original on arXiv Machine Learning →

arXiv:2602. 12124v2 Announce Type: replace Abstract: While most AI alignment research focuses on preventing models from generating explicitly harmful content, a more subtle risk arises from capability-seeking RL training in vulnerable environments.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Jun 16

Reward Hacking in Language Model Agents: Revisiting AI Safety Gridworlds

arXiv:2606. 15385v1 Announce Type: new Abstract: Reward hacking, where AI systems exploit misspecified objectives to achieve high reward without satisfying intended goals, remains a central challenge in AI safety.

By \"Omer Veysel \c{C}a\u{g}atan, Xuandong Zhao

llms agents reinforcement-learning safety

arXiv AI

Jun 19

The Autonomy Tax: Defense Training Breaks LLM Agents

arXiv:2603. 19423v2 Announce Type: replace-cross Abstract: Large language model (LLM) agents increasingly rely on external tools (file operations, API calls, database transactions) to autonomously complete complex multi-step tasks.

By Shawn Li, Yue Zhao

llms agents benchmarks safety

arXiv AI

Jun 11

Generalization Hacking: Models Can Game Reinforcement Learning by Preventing Behavioral Generalization

arXiv:2606. 12016v1 Announce Type: cross Abstract: Model post-training, and in particular reinforcement learning (RL), is one of the primary mechanisms by which developers can shape models' values and behaviors.

By Frank Xiao, Mary Phuong

reinforcement-learning fine-tuning

arXiv AI

Jun 2

Safety Alignment of LMs via Non-cooperative Games

arXiv:2512. 20806v3 Announce Type: replace Abstract: Ensuring the safety of language models (LMs) while maintaining their usefulness remains a critical challenge in AI alignment.

By Anselm Paulus, Ilia Kulikov, Brandon Amos, R\'emi Munos, Ivan Evtimov, Kamalika Chaudhuri, Arman Zharmagambetov

llms agents reinforcement-learning fine-tuning safety

arXiv Computation and Language

Oct 1

CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RL

arXiv:2609.39533v1 Announce Type: new Abstract: During reinforcement learning with verifiable rewards (RLVR), large language models (LLMs) can exploit loopholes in their environments to obtain high r...

By Shouli Wang, Yanfeng Jia, Zhihao Ou, Zitao Su, Ruize He, Haotong Xie, Hao Peng, Juanzi Li, Xiaozhi Wang

llms reinforcement-learning fine-tuning safety

arXiv Machine Learning

Jul 7

Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models

arXiv:2506. 07468v4 Announce Type: replace Abstract: Conventional large language model (LLM) safety alignment relies on a reactive, disjoint loop: attackers exploit a static model, then defenders patch exposed vulnerabilities.

By Mickel Liu, Liwei Jiang, Yancheng Liang, Simon Shaolei Du, Yejin Choi, Tim Althoff, Natasha Jaques

llms agents reinforcement-learning benchmarks safety

One email a morning, machine-written

Related stories

Reward Hacking in Language Model Agents: Revisiting AI Safety Gridworlds

The Autonomy Tax: Defense Training Breaks LLM Agents

Generalization Hacking: Models Can Game Reinforcement Learning by Preventing Behavioral Generalization

Safety Alignment of LMs via Non-cooperative Games

CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RL

Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models