arXiv AI October 19, 2026 By Jiawei Chen, Yi Zhang, Jiawei Du, Zhonghou Lv, Tairan Chen, Junyuan Shang, Jinchang Hou, Xiao Yang, Yu Tian, Chenfu Bao, Chao Yu, Zhaoxia Yin

Rethinking GRPO for Large Reasoning Model Safety Alignment

Read the original on arXiv AI →

The paper "Rethinking GRPO for Large Reasoning Model Safety Alignment" identifies two failure modes in Group Relative Policy Optimization—advantage collapse and unsafe diversification—that weaken safety learning in large reasoning models. It proposes the Beacon framework, which includes Safety Signal Recovery (SSR) and Safety-Utility Decoupled Alignment (SUDA), to restore effective safety signals and mitigate over-refusal. Experiments on Qwen3 and DeepSeek-R1-Distill models across eight benchmarks demonstrate that Beacon improves safety performance by up to 32.83 percentage points while keeping over-refusal low.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jun 30

Mitigating the Safety-utility Trade-off in LLM Alignment via Adaptive Safe Context Learning

arXiv:2602. 13562v2 Announce Type: replace-cross Abstract: While reasoning models have achieved remarkable success in complex reasoning tasks, their increasing power necessitates stringent safety measures.

By Yanbo Wang, Minzheng Wang, Jian Liang, Lu Wang, Yongcan Yu, Ran He

llms reinforcement-learning efficiency safety

arXiv AI

Jun 2

SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignment

arXiv:2606. 02530v1 Announce Type: new Abstract: Aligning Large Language Models (LLMs) with human values often degrades their general capabilities, termed the alignment tax.

By Hao Li, Jingkun An, Zijun Song, Pengyu Zhu, Rui Li, Hao Wang, Wendi Feng, Yesheng Liu, Lijun Li, Jin-Ge Yao, Lei Sha

llms efficiency benchmarks safety

arXiv AI

Sep 15

Beyond Safe Answers: Segment-Aware Listwise Alignment for Reasoning Safety in Large Reasoning Models

The paper introduces SaLT-DPO, a Segment‑aware Listwise Target DPO method designed to improve safety in Large Reasoning Models (LRMs). It decomposes responses into reasoning and answer segments, scores each segment’s safety, and aligns length‑normalized rewards with soft target distributions over multiple candidates. The approach also incorporates joint safety coherence regularization and utility anchoring to reduce unsafe content while maintaining compliance and reasoning performance.

By JungMin Yun, Junehyoung Kwon, Hayeong Ryu, Byeonggeuk Lim, Hoejoon Kwon, YoungBin Kim

reinforcement-learning benchmarks safety

arXiv AI

Sep 30

Towards Mitigating Deceptive Safety Alignment in Large Reasoning Models

arXiv:2609.36254v1 Announce Type: new Abstract: Large Reasoning Models (LRMs) are commonly trained with reinforcement learning (RL) to improve their generation of chain-of-thought (CoT) reasoning bef...

By Xiangyu Zhou, Saleh Zare Zade, Rafi Ibn Sultan, Alexander Kotov, Dongxiao Zhu

llms reinforcement-learning benchmarks safety

arXiv AI

Jul 7

Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Models

arXiv:2607. 02914v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated remarkable capabilities across diverse applications, yet ensuring their simultaneous safety, helpfulness, and trustworthiness remains a persistent challenge.

By Jiyang Guan, Yong Xie, Jun Chen, Jiexi Liu, Zipeng Ye, Defeng Li, Jiayu Shen, Jialing Tao, Hui Xue

llms reinforcement-learning fine-tuning benchmarks safety

arXiv Machine Learning

Jun 4

Few Tokens, Big Leverage: Preserving Safety Alignment by Constraining Safety Tokens during Fine-tuning

arXiv:2603. 07445v2 Announce Type: replace-cross Abstract: Large language models (LLMs) often require fine-tuning (FT) to perform well on downstream tasks, but FT can induce safety-alignment drift even when the training dataset contains only benign data.

By Guoli Wang, Haonan Shi, Tu Ouyang, An Wang

llms fine-tuning safety

One email a morning, machine-written

Related stories

Mitigating the Safety-utility Trade-off in LLM Alignment via Adaptive Safe Context Learning

SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignment

Beyond Safe Answers: Segment-Aware Listwise Alignment for Reasoning Safety in Large Reasoning Models

Towards Mitigating Deceptive Safety Alignment in Large Reasoning Models

Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Models

Few Tokens, Big Leverage: Preserving Safety Alignment by Constraining Safety Tokens during Fine-tuning