← Back to all news
arXiv Machine Learning By Bowen Fang (Columbia University), Dingmin Wang (AWS AI Lab), Hengzhi Pei (Amazon AGI)

$D^2$-OPD: Separating Direction and Distance in Extrapolative On-Policy Distillation

Read the original on arXiv Machine Learning →

$D^2$-OPD is a new on‑policy distillation method that separates the direction and distance of extrapolation when training a student model to match a teacher’s next‑token distributions. It introduces a per‑prefix hard KL constraint to bound the target’s distance from the teacher and an adaptive combination of the teacher’s post‑training shift with the initial student–teacher gap to set the extrapolation direction. Experiments on mathematical reasoning benchmarks show that $D^2$-OPD achieves the highest scores across three challenging settings, outperforming the strongest baseline by up to 9.45 points, with ablations confirming the value of each component.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Computation and Language
2d ago

Meet Students Where They Are: Student-Prefix Teacher Adaptation for On-Policy Distillation

The paper introduces Student‑Prefix Teacher Adaptation (SPTA), a method that trains a teacher model to better handle student‑generated prefixes before performing on‑policy distillation (OPD). By collecting prefixes of random lengths and using reinforcement learning with verifiable rewards (RLVR), the teacher learns to recover from student‑induced states rather than simply imitate the student’s reasoning. Experiments on mathematical reasoning benchmarks show that SPTA improves student performance compared to standard OPD and OPD with task‑level RLVR‑adapted teachers.

By Xue Zhang, Songming Zhang, Yunlong Liang, Fandong Meng, Yufeng Chen, Jinan Xu
reinforcement-learningefficiencybenchmarkssafety
More like this →
arXiv AI
Oct 7

E$^2$-OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation

E$^2$-OPSD tackles a failure mode in on‑policy self‑distillation where the student’s token entropy surpasses the teacher’s, a problem called entropy overshoot. The method replaces the teacher’s answer with a retrieved solved neighboring problem to provide transferable reasoning guidance and uses the student‑teacher entropy gap to adjust token corrections. These changes improve math reasoning scores by up to 4.3 points in mean@16 and yield out‑of‑domain gains of up to 4.9 points in mean@16 and 5.5 points in pass@8, all without extra forward passes or networks.

By Yifei Liu, Minghao Fang, Xinyu Gu, Chengkai Yao, Mengdi Liu, Tengfei Ma, Jiangbin Zheng, Chang Yu, Zhangyang Gao
efficiency
More like this →
arXiv Machine Learning
Oct 16

Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation

Waypoint-OPD is a new on-policy distillation method that leverages an intermediate teacher checkpoint to compare early and late policy shifts in reinforcement learning. By gating the endpoint shift based on the directional agreement between these early and late shifts, the method suppresses conflicting updates and requires only one additional frozen-teacher scoring pass. Experiments with Qwen3-4B teachers and larger Qwen3 student models in mathematical and code reasoning tasks show that Waypoint-OPD consistently outperforms endpoint-only transfer across all six domain–student settings.

By Bingxi Hou, Guochao Jiang, Guofeng Quan, Weiqing Li, Wenfeng Feng, Guohua Liu, Yuewei Zhang
reinforcement-learningefficiency
More like this →