arXiv:2606. 03089v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD) has emerged as an efficient post-training paradigm by using a teacher conditioned on privileged information to provide dense token-level supervision.
arXiv:2607. 02502v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access.
EOPSA (Efficient On-Policy Self-Distilled Safety Alignment) addresses inefficiencies in On-Policy Self-Distillation (OPSD) for safety alignment by focusing training on safety-critical tokens. It introduces Adaptive Rollout Scheduling, which limits generation length based on a Teacher Rescue Rate metric, and Selective Distillation, which filters out safety-neutral tokens to concentrate gradient updates on safety-pivotal transitions. Experiments on models up to 32B parameters show that EOPSA reduces rollout computation by about 50% and backpropagates through only roughly 2% of tokens, outperforming full-token distillation baselines in safety compliance and reasoning retention.
The paper investigates on‑policy self‑distillation (OPSD) as a method to enhance reasoning in language models, focusing on mathematical reasoning across models from 0.6B to 8B parameters. Through controlled experiments and token‑level analysis, the authors find that OPSD’s effectiveness depends on alignment between the teacher’s reasoning mode and the full teacher prefix, rather than on privileged semantics alone. They observe that OPSD only improves reasoning in limited compatibility regimes, while often causing length growth, degradation, or behavioral collapse, and that the teacher’s signal is unstable and not predictive of downstream performance.
arXiv:2605. 12400v2 Announce Type: replace-cross Abstract: We study on-policy self-distillation (OPSD), where a language model improves its reasoning ability by distilling privileged teacher distributions along its own on-policy trajectories.
arXiv:2606. 08432v1 Announce Type: new Abstract: On-policy distillation (OPD) has become a central post-training tool for large language models (LLMs), providing dense per-token teacher supervision along the student's own rollouts.
On-Policy distillation (OPD) transfers teacher capabilities by supervising student-sampled trajectories with dense token-level teacher signals. Recent selective OPD methods improve this process by prioritizing signals that are confident, informative, or learnable.
arXiv:2608. 03632v1 Announce Type: new Abstract: On-Policy distillation (OPD) transfers teacher capabilities by supervising student-sampled trajectories with dense token-level teacher signals.
The paper introduces COLAGUARD, a guardrail model that embeds multi-step safety reasoning into a continuous latent space, allowing efficient hidden-state propagation during inference. Compared to existing methods, COLAGUARD achieves an 8.24‑point macro‑F1 improvement over Llama Guard 3 and matches the explicit reasoning baseline GuardReasoner, while delivering a 12.9× speedup and a 22.4× reduction in token usage across ten moderation settings and eight safety benchmarks.
SOD: Step-wise On-policy Distillation for Small Language Model Agents proposes a new framework that adaptively reweights distillation strength at each reasoning step based on step-level divergence. This approach mitigates cascading errors in tool-integrated reasoning by attenuating misleading teacher signals in high-divergence regions while preserving dense guidance where student and teacher align. Experiments on math, science, and code benchmarks show up to 20.86% improvement over the second-best baseline, with a 0.6B student scoring 26.13% on AIME 2025.
arXiv:2609.37132v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) improves large language models by letting a self-teacher with privileged information provide dense token-level super...
The paper introduces On-Policy Attention Self-Distillation (OPASD), a method that augments token-level supervision with solution-conditioned attention distillation for reasoning models. OPASD projects a privileged teacher’s attention onto student-visible positions, renormalizes the distribution, and aligns it with the student. Experiments on three model sizes and four math benchmarks show that OPASD improves accuracy by 4.98–8.40 percentage points, reduces generated tokens by 73.9%, cuts compute by 72.6%, and trains 1.53× faster compared to token-only distillation.
Contrastive On-Policy Distillation (COPD) is a framework that improves on-policy distillation by using a frozen teacher to evaluate student states under two contrasting prompts—one encouraging low reasoning effort and one encouraging high effort. The difference in log‑probabilities between these prompts provides a token‑level advantage signal that guides the student toward more concise and efficient reasoning strategies. Experiments on nine multimodal benchmarks show that COPD reduces reasoning length while maintaining task performance, and the contrastive approach can also be applied to on‑policy self‑distillation, allowing a model to compress its own reasoning without an external teacher.