← Back to all news
arXiv AI By Yizheng Yang, Haining Yu, Yuechen Wang, Yikai Hou, Xing Fu, Jinbo Yang, Tianqing Zhu

First Token Matters: Understanding Safety Collapse in Large Reasoning Models

Read the original on arXiv AI →

The paper investigates why large reasoning models (LRMs) lose safety alignment when faced with harmful queries. By analyzing token-level refusal dynamics, the authors identify a vulnerability called Onset Refusal Collapse (ORC), where the refusal signal drops sharply at the first generated token, leading to unsafe responses. They introduce SafeToken, a lightweight inference-time intervention that injects a learned safety anchor at reasoning onset, which mitigates ORC, improves safety on harmful-query benchmarks, and largely preserves reasoning utility.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 15

Beyond Safe Answers: Segment-Aware Listwise Alignment for Reasoning Safety in Large Reasoning Models

The paper introduces SaLT-DPO, a Segment‑aware Listwise Target DPO method designed to improve safety in Large Reasoning Models (LRMs). It decomposes responses into reasoning and answer segments, scores each segment’s safety, and aligns length‑normalized rewards with soft target distributions over multiple candidates. The approach also incorporates joint safety coherence regularization and utility anchoring to reduce unsafe content while maintaining compliance and reasoning performance.

By JungMin Yun, Junehyoung Kwon, Hayeong Ryu, Byeonggeuk Lim, Hoejoon Kwon, YoungBin Kim
reinforcement-learningbenchmarkssafety
More like this →
arXiv AI
Aug 26

TRACE: An Evidence-Grounded Benchmark for Safety Evaluation of Large Reasoning Models

TRACE is a new benchmark that evaluates the safety of Large Reasoning Models (LRMs) across the entire inference pipeline, including prompts, reasoning traces, and final responses. It provides prompts in two languages covering nine risk categories and ten attack strategies, and for each prompt four LRMs generate traces and responses that are annotated for safety with supporting evidence extracted from the source text. Evaluation of 18 guardrail models on TRACE shows that detecting unsafe content in reasoning traces is much harder than in prompts or final responses, and that current models struggle to extract the necessary evidence.

By Zhenyu Wu, Siyuan Chen, Changchun Yang, Jiaqi Dong, Min Zhou, Ali Almadan, Talal Hammad, Faisal Wahbo, Aminullah Tora, Mona Alshahrani, Xin Gao
benchmarkssafety
More like this →
arXiv AI
3d ago

GUARD: Natural Forgetting in Large Reasoning Models via Guided Answer-Reasoning Distillation

The paper introduces GUARD, a method for natural forgetting in large reasoning models that transforms unsafe disclosures into safe-exit trajectories using guided answer‑reasoning distillation. It aligns a frozen model with guidance tokens and distills this behavior into the parameters, aiming for a coherent, non‑disclosing chain of thought followed by a refusal‑style answer. The authors also propose the Natural Forgetting Reasoning Score (NFRS) to evaluate structural stability, fluency, and unsupported substitutes, and demonstrate GUARD’s effectiveness on R‑TOFU and a STAR‑1‑derived harmful‑intent setting.

By Zeyu Yan, Guanghao Zhou, Minghui Qiu, Ming Gao, Cen Chen
efficiencysafety
More like this →