arXiv Machine Learning
Reinforcing What the Student Has Learned: Teacher-Guided On-Policy Distillation
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Computation and Language
2d ago
Meet Students Where They Are: Student-Prefix Teacher Adaptation for On-Policy Distillation
The paper introduces Student‑Prefix Teacher Adaptation (SPTA), a method that trains a teacher model to better handle student‑generated prefixes before performing on‑policy distillation (OPD). By collecting prefixes of random lengths and using reinforcement learning with verifiable rewards (RLVR), the teacher learns to recover from student‑induced states rather than simply imitate the student’s reasoning. Experiments on mathematical reasoning benchmarks show that SPTA improves student performance compared to standard OPD and OPD with task‑level RLVR‑adapted teachers.
arXiv Machine Learning
Oct 16
Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation
Waypoint-OPD is a new on-policy distillation method that leverages an intermediate teacher checkpoint to compare early and late policy shifts in reinforcement learning. By gating the endpoint shift based on the directional agreement between these early and late shifts, the method suppresses conflicting updates and requires only one additional frozen-teacher scoring pass. Experiments with Qwen3-4B teachers and larger Qwen3 student models in mathematical and code reasoning tasks show that Waypoint-OPD consistently outperforms endpoint-only transfer across all six domain–student settings.
Hugging Face Trending Papers
Sep 29
Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models
On-policy distillation (OPD) trains a student on its own generated responses using dense, token-level supervision from a stronger teacher. Vanilla OPD treats all teacher signals equally, assuming that...
arXiv Machine Learning
Jul 17
On-Policy Delta Distillation
arXiv:2607. 15161v1 Announce Type: new Abstract: On-policy distillation is an alternative post-training method in reinforcement learning that alleviates the constraints imposed by reward models by providing token-level supervision from a teacher model.
arXiv AI
Sep 21
Calibrating Teacher--Student Discrepancy for On-Policy Distillation
arXiv:2609.21619v1 Announce Type: new Abstract: On-policy distillation (OPD) improves reasoning models by learning the token-level discrepancy between a stronger teacher and an on-policy student. How...
arXiv Machine Learning
21h ago
$D^2$-OPD: Separating Direction and Distance in Extrapolative On-Policy Distillation
$D^2$-OPD is a new on‑policy distillation method that separates the direction and distance of extrapolation when training a student model to match a teacher’s next‑token distributions. It introduces a per‑prefix hard KL constraint to bound the target’s distance from the teacher and an adaptive combination of the teacher’s post‑training shift with the initial student–teacher gap to set the extrapolation direction. Experiments on mathematical reasoning benchmarks show that $D^2$-OPD achieves the highest scores across three challenging settings, outperforming the strongest baseline by up to 9.45 points, with ablations confirming the value of each component.