On-policy distillation (OPD) provides teacher supervision on states visited by the student, reducing the distribution gap between training and inference. However, in multi-turn agentic tasks, student deviations may accumulate over time, gradually moving the trajectory away from states where teacher guidance remains effective.
arXiv:2608. 01953v1 Announce Type: cross Abstract: On-policy distillation (OPD) provides teacher supervision on states visited by the student, reducing the distribution gap between training and inference.
State-OPD introduces a state‑grounded on‑policy distillation framework that separates the objective task and action‑observation record from the student’s internal reasoning trajectory. By removing prior student reasoning and resampling complete reasoning‑action responses at high‑entropy states, the method improves teacher supervision and action diversity. Evaluations on ALFWorld, ScienceWorld, and WebShop show State‑OPD achieves the highest macro‑average success rates for Qwen3‑1.7B and Qwen3‑4B students, even surpassing a larger teacher model on certain tasks.
arXiv:2606. 15912v1 Announce Type: cross Abstract: Multi-turn agents that plan, invoke tools, and interact with environments offer a promising paradigm for solving complex tasks, yet their capabilities typically rely on very large models whose inference cost is prohibitive in practice.
arXiv:2609.37522v1 Announce Type: new Abstract: On-policy distillation (OPD) trains compact language agents with teacher feedback on student-generated trajectories. In multi-turn tasks, compounding e...
arXiv:2608. 16333v1 Announce Type: cross Abstract: On-policy distillation (OPD) aligns a student model with a teacher's logit distribution on student-generated trajectories.
On-policy distillation (OPD) aligns a student model with a teacher's logit distribution on student-generated trajectories. This approach has achieved strong empirical gains and can often surpass conve...
arXiv:2610.19145v1 Announce Type: new Abstract: On-policy distillation (OPD) is increasingly studied as a cheaper alternative to reinforcement learning, yet it remains a form of knowledge distillatio...
Branching On-Policy Distillation (BranchOPD) enhances on‑policy distillation by selecting informative decision points based on student uncertainty and teacher‑student disagreement, resampling continuations from shared prefixes, and applying token‑level teacher supervision to both main trajectories and branches. This approach reduces student‑generated token costs by about 19% on ALFWorld and 31% on WebShop compared to seven independent trajectories, and introduces protocol‑boundary repair (PBR) to mitigate repetition under protocol constraints. BranchOPD outperforms Vanilla OPD and Multi‑Rollout OPD across benchmarks, achieving significant gains in mathematics accuracy and ALFWorld success rates, with mechanistic analyses showing that branching at uncertain decisions expands behavior coverage.
arXiv:2609.14636v1 Announce Type: new Abstract: On-policy distillation (OPD) has become a standard approach for transferring capabilities from large teachers to compact students. Its cost, however, i...
The paper introduces STI-OPD, a stochastic teacher intervention framework that improves on-policy distillation for multi-turn agentic tasks. By measuring teacher‑student policy discrepancy with KL divergence, STI-OPD probabilistically replaces student actions with teacher actions to provide reliable supervision. An Importance‑Weighted Reverse KL objective then corrects token sampling mismatches, enabling the student to learn from mixed‑policy trajectories and outperform prior OPD baselines across benchmarks.
The paper introduces Advantage-Based On-Policy Distillation (AOPD), a method that selectively weights teacher actions during online policy distillation by their counterfactual advantage to the student. Unlike standard OPD, which follows the teacher at every state, AOPD reinforces only those teacher actions that improve the student's expected outcome, using an estimator that relies solely on standard student rollouts and terminal rewards. Experiments on math, coding, and Medqa tasks demonstrate that AOPD outperforms both distillation and reinforcement learning baselines in external-teacher and self-distillation settings.