arXiv:2606. 24143v1 Announce Type: new Abstract: On-policy distillation (OPD) trains a student on its own rollouts guided by teacher feedback and is becoming increasingly important for large language model (LLM) post-training.
arXiv:2605. 03677v2 Announce Type: replace Abstract: On-policy distillation (OPD) has recently emerged as an effective post-training paradigm for consolidating the capabilities of specialized expert models into a single student model.
The paper investigates the learning dynamics of on‑policy distillation (OPD), a method that combines teacher supervision with training on student‑generated trajectories. It identifies two stages of the feedback loop: converting the distillation signal into parameter updates, and broadcasting those updates to alter predictions and reshape the rollout distribution. The study explains why students may be unresponsive to teacher disagreement and how updates that improve local agreement can lead to degeneration by increasing the likelihood of poor continuations. It also shows that offline distillation can prepare a student for OPD without boosting immediate accuracy, yet OPD can still yield significant accuracy gains after sufficient offline preparation.
SOD: Step-wise On-policy Distillation for Small Language Model Agents proposes a new framework that adaptively reweights distillation strength at each reasoning step based on step-level divergence. This approach mitigates cascading errors in tool-integrated reasoning by attenuating misleading teacher signals in high-divergence regions while preserving dense guidance where student and teacher align. Experiments on math, science, and code benchmarks show up to 20.86% improvement over the second-best baseline, with a 0.6B student scoring 26.13% on AIME 2025.
arXiv:2609.37500v1 Announce Type: new Abstract: On-policy distillation (OPD) trains language models using dense token-level teacher supervision on student-generated trajectories. However, its relianc...
The paper introduces Un-OPD, an unbiased on‑policy distillation framework for block diffusion language models that addresses two key optimization biases: context misalignment from mismatched block boundaries and an intrinsic bias that causes overconfidence collapse. Un-OPD employs a boundary‑aware step filtering strategy to remove misaligned decoding steps and a support‑rebalanced confidence calibration to moderate optimization intensity at high‑support positions. The method also includes a rollout reuse mechanism to cut generation overhead, and experiments on math reasoning and code generation tasks demonstrate stable training, improved performance, and roughly 50% faster wall‑clock time.
RollVerify is a lightweight reinforcement learning framework that addresses the trade‑off between efficiency and accuracy in long‑tail rollout settings. It introduces an off‑policy shift metric (OPS) to quantify deviation in partially generated trajectories and uses sequence‑level and token‑level verification to truncate invalid suffixes before training. Experiments on mathematical reasoning tasks show that RollVerify matches on‑policy accuracy while cutting training costs, with preliminary evidence from code‑generation tasks.
arXiv:2605.07865v2 Announce Type: replace-cross Abstract: On-Policy Distillation (OPD) has emerged as a dominant post-training paradigm for Large Language Models, especially in reasoning domains. How...
arXiv:2608. 07935v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) adapts a language model by distilling guidance from a frozen teacher on trajectories sampled from the student.
ThunderSyncRL is a training framework that eliminates idle time in agentic reinforcement learning by starting gradient computation immediately once all necessary inputs are available, thereby avoiding policy staleness. It applies to group relative policy optimization (GRPO) by computing trajectory score gradients as soon as rewards arrive, and to on‑policy distillation (OPD) by updating gradients for completed agentic turns while tool calls execute. Experiments on SWE‑bench Verified and Terminal Bench 4.0 show that ThunderSyncRL matches synchronous training performance up to 1.9× faster and outperforms asynchronous training by up to 2.47 percentage points at a fixed budget.
arXiv:2608. 03632v1 Announce Type: new Abstract: On-Policy distillation (OPD) transfers teacher capabilities by supervising student-sampled trajectories with dense token-level teacher signals.
arXiv:2610.19439v1 Announce Type: new Abstract: While LLM-based agent systems have demonstrated remarkable proficiency in reasoning and solving complex real-world tasks, they also come with substanti...