arXiv AI
2d ago

SAVeR$^2$: Reasoning-based Safety Alignment for Large Reasoning Models via Verifiable Rewards

SAVeR$^2$ is a safety alignment method for Large Reasoning Models that decomposes safety objectives into four fine‑grained verifiable reward signals and uses a difficulty‑aware data selection strategy to stabilize training. It introduces a reasoning‑preserving gradient projection mechanism that resolves conflicts between safety and reasoning gradients, eliminating the safety tax. Experiments on models from 1.5B to 671B show that SAVeR$^2$ improves safety performance and reduces over‑refusal rates without harming general reasoning abilities.

By Yichen Sun, Qirui Liu, Yan Wang, Jianan Lin, Linbo Jiang, Shiyu Wang, Zhibo Wang, Zhixuan Chu