arXiv AI
Mitigating Reasoning-Induced Misalignment via Safety-Direction Penalty
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
Hugging Face Trending Papers
Aug 24
Mitigating Reasoning-Induced Misalignment via Safety-Direction Penalty
Reasoning-Induced Misalignment, where fine-tuning on reasoning data containing no harmful content, including mathematics, code, and problem-solving with chain-of-thought traces can induce harmful beha...
arXiv AI
2d ago
SAVeR$^2$: Reasoning-based Safety Alignment for Large Reasoning Models via Verifiable Rewards
SAVeR$^2$ is a safety alignment method for Large Reasoning Models that decomposes safety objectives into four fine‑grained verifiable reward signals and uses a difficulty‑aware data selection strategy to stabilize training. It introduces a reasoning‑preserving gradient projection mechanism that resolves conflicts between safety and reasoning gradients, eliminating the safety tax. Experiments on models from 1.5B to 671B show that SAVeR$^2$ improves safety performance and reduces over‑refusal rates without harming general reasoning abilities.
arXiv AI
Oct 21
Rethinking LLM Fine-Tuning via Weight Space Reparameterization: Preserving Safety during Downstream Adaptation
arXiv:2610.26443v2 Announce Type: replace-cross Abstract: Fine-tuning large language models (LLMs) on downstream tasks often weakens their previously aligned safety behavior, creating an inherent tra...
Hugging Face Trending Papers
Sep 28
EOPSA: Efficient On-Policy Self-Distilled Safety Alignment
EOPSA (Efficient On-Policy Self-Distilled Safety Alignment) addresses inefficiencies in On-Policy Self-Distillation (OPSD) for safety alignment by focusing training on safety-critical tokens. It introduces Adaptive Rollout Scheduling, which limits generation length based on a Teacher Rescue Rate metric, and Selective Distillation, which filters out safety-neutral tokens to concentrate gradient updates on safety-pivotal transitions. Experiments on models up to 32B parameters show that EOPSA reduces rollout computation by about 50% and backpropagates through only roughly 2% of tokens, outperforming full-token distillation baselines in safety compliance and reasoning retention.
arXiv AI
Oct 27
EOPSA: Efficient On-Policy Self-Distilled Safety Alignment
arXiv:2609.34519v2 Announce Type: replace Abstract: On-Policy Self-Distillation (OPSD) has emerged as a promising paradigm for safety alignment, delivering dense, token-level supervision by distillin...
arXiv AI
Aug 24
RefusalGuard: Geometry-Preserving Fine-Tuning for Safety in LLMs
arXiv:2605.01913v2 Announce Type: replace-cross Abstract: Fine-tuning safety-aligned language models for downstream tasks often leads to substantial degradation of refusal behavior, making models vul...