← Back to all news
arXiv AI By Langlin Huang, Hao Liu, Mononito Goswami, Xinyu Li, Prithwith Jana, Nikos Kanakaris, Patrick Bl\"obaum, Purak Jain

On the Off-Policy Teacher in On-Policy Distillation

Read the original on arXiv AI →

The paper introduces SCOUT, a co‑training framework that adapts an off‑policy teacher to better continue from student‑generated prefixes in on‑policy distillation (OPD). By periodically optimizing the teacher’s conditional continuation ability using reinforcement learning with verifiable rewards, SCOUT improves the teacher’s performance on student prefixes. Experiments across various teacher‑student setups, model scales, and reasoning domains show that SCOUT consistently enhances the effectiveness of OPD.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computation and Language
6d ago

Recursive Self-Improvement via On-Policy Distillation for Reasoning

The paper introduces a recursive self-improvement framework for language models that replaces an external teacher with a frozen copy of the student, enabling dynamic co-evolution (DCE) and self-refined concise learning (SRCL). DCE allows the privileged teacher to evolve alongside the student, while SRCL trains on shorter, verified rewrites to reduce verbosity. Experiments show that the combined DCE+SRCL approach outperforms traditional on‑policy self‑distillation across multiple model sizes and math benchmarks, achieving significant accuracy gains and shorter outputs.

By Shangjian Yin, Zehao Zhao, Kavosh Asadi, Rui Liu, Yuchen Lu, Shike Mei, Hang Cui, Luke Simon, Zhouxing Shi, Hamed Firooz
efficiencybenchmarks
More like this →
arXiv Machine Learning
Sep 22

Teacher Should Think Ahead: Adaptive Continuations for Reliable On-Policy Distillation

Teacher Should Think Ahead: Adaptive Continuations for Reliable On-Policy Distillation explores how on‑policy distillation (OPD) can be improved by addressing teacher uncertainty that contracts as the teacher continues from a student‑generated prefix. The authors identify Teacher Uncertainty Contraction (TUC) and theoretically analyze its variance‑bias trade‑off, leading to the proposal of Adaptive‑Continuations On‑Policy Distillation (AC‑OPD). Experiments on mathematical reasoning and code generation show that AC‑OPD consistently outperforms standard OPD, with controlled‑continuation and matched‑budget analyses supporting the adaptive‑continuation design.

By Jingang Zhou, Yuyi Zhou, Haiyang Guo, Xukai Wang, Shuai Feng, Sirui Gao, Jian Xu, Qingpei Guo, Xu-Yao Zhang
llmsefficiencysafety
More like this →