The paper proposes Mixture Proposal On-Policy Distillation (MPOPD) to address a mismatch issue in on‑policy distillation (OPD) where the rollout policy is fixed to the current student. By treating the rollout policy as an optimization target, MPOPD derives a geometric mixture of student and teacher policies and introduces a proposal policy for efficient rollout generation. Experiments on 16 reasoning benchmarks show consistent gains over OPD, notably a 6.91% accuracy improvement on math reasoning with Qwen3-4B-Base and a 41% reduction in generation length.
Offline on-policy distillation gains efficiency by collecting student trajectories and teacher supervision once and reusing them throughout optimization. The same reuse makes imperfect supervision per...
arXiv:2606. 21994v2 Announce Type: replace Abstract: On-policy distillation (OPD) improves reasoning models by applying dense teacher supervision on student-sampled trajectories.
The paper introduces a method for offline on‑policy distillation that addresses the problem of imperfect teacher supervision. By training on teacher‑successful problems and measuring changes in token likelihoods on teacher‑failed trajectories, the authors derive a learnability signal that weights the distillation loss. This approach improves performance on mathematical reasoning and code generation tasks while reducing computational cost compared to online distillation.
arXiv:2609.36546v1 Announce Type: cross Abstract: On-policy distillation (OPD) trains a student model on its self-generated trajectories with dense token-level teacher feedback. However, naive OPD ma...
On-policy distillation (OPD) aligns a student model with a teacher's logit distribution on student-generated trajectories. This approach has achieved strong empirical gains and can often surpass conve...
The paper introduces Success-Referenced On-Policy Distillation (SR-OPD), a method that reduces teacher computation in on-policy distillation by selectively providing teacher supervision only for prompts where the student has both successful and failed rollouts. SR-OPD uses successful rollouts as references to prioritize failed rollouts that diverge significantly from the successful ones, while also considering estimated teacher-input cost. Experiments on three teacher-student pairs across six mathematical reasoning benchmarks show that SR-OPD requires only 3.46‑5.02% of the teacher-input tokens of Vanilla OPD in a one-pass setting, yet maintains comparable reasoning performance, and further validates its design choices under a 5% teacher-input budget.
arXiv:2608. 16333v1 Announce Type: cross Abstract: On-policy distillation (OPD) aligns a student model with a teacher's logit distribution on student-generated trajectories.
arXiv:2606. 08432v1 Announce Type: new Abstract: On-policy distillation (OPD) has become a central post-training tool for large language models (LLMs), providing dense per-token teacher supervision along the student's own rollouts.
The paper introduces Advantage-Based On-Policy Distillation (AOPD), a method that selectively weights teacher actions during online policy distillation by their counterfactual advantage to the student. Unlike standard OPD, which follows the teacher at every state, AOPD reinforces only those teacher actions that improve the student's expected outcome, using an estimator that relies solely on standard student rollouts and terminal rewards. Experiments on math, coding, and Medqa tasks demonstrate that AOPD outperforms both distillation and reinforcement learning baselines in external-teacher and self-distillation settings.
The paper introduces SCOUT, a co‑training framework that adapts an off‑policy teacher to better continue from student‑generated prefixes in on‑policy distillation (OPD). By periodically optimizing the teacher’s conditional continuation ability using reinforcement learning with verifiable rewards, SCOUT improves the teacher’s performance on student prefixes. Experiments across various teacher‑student setups, model scales, and reasoning domains show that SCOUT consistently enhances the effectiveness of OPD.
The paper introduces STI-OPD, a stochastic teacher intervention framework that improves on-policy distillation for multi-turn agentic tasks. By measuring teacher‑student policy discrepancy with KL divergence, STI-OPD probabilistically replaces student actions with teacher actions to provide reliable supervision. An Importance‑Weighted Reverse KL objective then corrects token sampling mismatches, enabling the student to learn from mixed‑policy trajectories and outperform prior OPD baselines across benchmarks.
arXiv:2609.37170v1 Announce Type: cross Abstract: Off-policy and on-policy distillation have traditionally been formulated as separate paradigms, each favoring a different property of distillation tr...