SAVeR$^2$ is a safety alignment method for Large Reasoning Models that decomposes safety objectives into four fine‑grained verifiable reward signals and uses a difficulty‑aware data selection strategy to stabilize training. It introduces a reasoning‑preserving gradient projection mechanism that resolves conflicts between safety and reasoning gradients, eliminating the safety tax. Experiments on models from 1.5B to 671B show that SAVeR$^2$ improves safety performance and reduces over‑refusal rates without harming general reasoning abilities.
By Yichen Sun, Qirui Liu, Yan Wang, Jianan Lin, Linbo Jiang, Shiyu Wang, Zhibo Wang, Zhixuan Chu
Large reasoning models generate chain-of-thought (CoT) before answering but struggle with safety alignment and can be misled by flawed premises. The paper introduces RECAP, a reinforcement learning approach that trains models to override flawed reasoning paths and produce safe, helpful responses without extra training cost. RECAP improves safety, jailbreak robustness, and reduces overrefusal while preserving core reasoning abilities and inference token budget.
By ShengYun Peng, Pin-Yu Chen, Eric Smith, Song Jiang, Hongyuan Zhan, Haozhu Wang, Mahesh Pasupuleti, Duen Horng Chau, Jianfeng Chi
arXiv:2606. 31748v1 Announce Type: new Abstract: Safety training on language models often induces over-refusal: improved safety on harmful prompts at the cost of increased refusal on harmless ones.
By Taeyoun Kim, Aviral Kumar
The paper introduces "cunning questions"—non‑safety prompts that contain misleading premises or subtle inconsistencies—to train large language models (LLMs) to scrutinize underlying intent and assumptions. Experiments show that incorporating these questions improves robustness against out‑of‑distribution jailbreak attacks and enhances subsequent safety fine‑tuning, achieving a new state‑of‑the‑art reduction in mean ASR from 17.40% to 15.05% across nine backbone–benchmark combinations. The authors argue that this training fosters vigilance, enabling models to prioritize safety judgments before engaging in harmful planning.
By Youjia Wang, Lin Xu, Yang Sun, Yuxiao Lu, Chengfang Fang, Jie Shi
arXiv:2603. 26846v2 Announce Type: replace-cross Abstract: As Large Language Models (LLMs) expand in capability and application scope, their trustworthiness becomes critical.
By Guoxi Zhang, Jiawei Chen, Tianzhuo Yang, Lang Qin, Juntao Dai, Yaodong Yang, Jingwei Yi
The paper introduces SaLT-DPO, a Segment‑aware Listwise Target DPO method designed to improve safety in Large Reasoning Models (LRMs). It decomposes responses into reasoning and answer segments, scores each segment’s safety, and aligns length‑normalized rewards with soft target distributions over multiple candidates. The approach also incorporates joint safety coherence regularization and utility anchoring to reduce unsafe content while maintaining compliance and reasoning performance.
By JungMin Yun, Junehyoung Kwon, Hayeong Ryu, Byeonggeuk Lim, Hoejoon Kwon, YoungBin Kim
arXiv:2607. 02914v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated remarkable capabilities across diverse applications, yet ensuring their simultaneous safety, helpfulness, and trustworthiness remains a persistent challenge.
By Jiyang Guan, Yong Xie, Jun Chen, Jiexi Liu, Zipeng Ye, Defeng Li, Jiayu Shen, Jialing Tao, Hui Xue
arXiv:2608. 09542v1 Announce Type: cross Abstract: Large reasoning models (LRMs) achieve remarkable success on complex tasks but remain vulnerable to harmful prompts that induce unsafe outputs.
By Hongli Shen, Shaopeng Fu, Qinbo Zhang, Jian Li, Di Wang
The paper "Rethinking GRPO for Large Reasoning Model Safety Alignment" identifies two failure modes in Group Relative Policy Optimization—advantage collapse and unsafe diversification—that weaken safety learning in large reasoning models. It proposes the Beacon framework, which includes Safety Signal Recovery (SSR) and Safety-Utility Decoupled Alignment (SUDA), to restore effective safety signals and mitigate over-refusal. Experiments on Qwen3 and DeepSeek-R1-Distill models across eight benchmarks demonstrate that Beacon improves safety performance by up to 32.83 percentage points while keeping over-refusal low.
By Jiawei Chen, Yi Zhang, Jiawei Du, Zhonghou Lv, Tairan Chen, Junyuan Shang, Jinchang Hou, Xiao Yang, Yu Tian, Chenfu Bao, Chao Yu, Zhaoxia Yin
arXiv:2608. 03745v1 Announce Type: new Abstract: Chain-of-Thought (CoT) reasoning offers a promising window into model monitoring.
By Dominik Meier, Luca Joshua Francis, Marco Bernhard Kaiser, Terry Ruas, Jan Philip Wahle, Bela Gipp
The paper introduces Continuous Adversarial Training for Consistent Honesty (CATCH), a method that trains large language models to produce honest responses while preserving their overall capabilities. Using only a few hundred self‑reporting scenarios, CATCH is applied to Llama‑3.3‑70B‑Instruct and Qwen3‑3.6‑27B, and the resulting models show improved honesty across multiple benchmarks, including MASK, Among Us, and AuditBench. Compared to prompting, supervised fine‑tuning, and state‑of‑the‑art honesty steering, CATCH achieves superior honesty while maintaining utility through a novel choice of utility data.
By Yuan-Chen Chang, Niklas Herbster, Gauthier Gidel, Tommaso Tosato
Large Reasoning Models (LRMs) use explicit chain‑of‑thought reasoning and large context windows to perform complex tasks, but these features create new attack surfaces. The paper introduces SRCF, an attack that steers LRMs by prepending counter‑aligned few‑shot conversations with explicit CoT traces, causing unsafe outputs on harmful queries and unwarranted refusals on benign ones, without needing model internals. To counter this, the authors propose ARCF, a post‑training defense that exposes models to counter‑aligned conversational contexts while enforcing aligned targets, improving safety and helpfulness without harming utility.
By Xiangyu Zhou, Saleh Zare Zade, Dongxiao Zhu