arXiv AI
$\boldsymbol{f}$-OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Control
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jun 24
AsyncOPD: How Stale Can On-Policy Distillation Be?
arXiv:2606. 24143v1 Announce Type: new Abstract: On-policy distillation (OPD) trains a student on its own rollouts guided by teacher feedback and is becoming increasingly important for large language model (LLM) post-training.
arXiv Machine Learning
Jul 7
Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe
arXiv:2605. 03677v2 Announce Type: replace Abstract: On-policy distillation (OPD) has recently emerged as an effective post-training paradigm for consolidating the capabilities of specialized expert models into a single student model.
arXiv Machine Learning
1d ago
Learning Dynamics of On-Policy Distillation
The paper investigates the learning dynamics of on‑policy distillation (OPD), a method that combines teacher supervision with training on student‑generated trajectories. It identifies two stages of the feedback loop: converting the distillation signal into parameter updates, and broadcasting those updates to alter predictions and reshape the rollout distribution. The study explains why students may be unresponsive to teacher disagreement and how updates that improve local agreement can lead to degeneration by increasing the likelihood of poor continuations. It also shows that offline distillation can prepare a student for OPD without boosting immediate accuracy, yet OPD can still yield significant accuracy gains after sufficient offline preparation.
arXiv AI
Aug 19
SOD: Step-wise On-policy Distillation for Small Language Model Agents
SOD: Step-wise On-policy Distillation for Small Language Model Agents proposes a new framework that adaptively reweights distillation strength at each reasoning step based on step-level divergence. This approach mitigates cascading errors in tool-integrated reasoning by attenuating misleading teacher signals in high-divergence regions while preserving dense guidance where student and teacher align. Experiments on math, science, and code benchmarks show up to 20.86% improvement over the second-best baseline, with a 0.6B student scoring 26.13% on AIME 2025.
arXiv Machine Learning
Sep 30
REVO: Rollout-Efficient Off-Policy Distillation via Variance-Guided Reuse
arXiv:2609.37500v1 Announce Type: new Abstract: On-policy distillation (OPD) trains language models using dense token-level teacher supervision on student-generated trajectories. However, its relianc...
arXiv Computation and Language
Oct 16
Towards Unbiased On-Policy Distillation for Block Diffusion Language Models
The paper introduces Un-OPD, an unbiased on‑policy distillation framework for block diffusion language models that addresses two key optimization biases: context misalignment from mismatched block boundaries and an intrinsic bias that causes overconfidence collapse. Un-OPD employs a boundary‑aware step filtering strategy to remove misaligned decoding steps and a support‑rebalanced confidence calibration to moderate optimization intensity at high‑support positions. The method also includes a rollout reuse mechanism to cut generation overhead, and experiments on math reasoning and code generation tasks demonstrate stable training, improved performance, and roughly 50% faster wall‑clock time.