arXiv Machine Learning

REVO: Rollout-Efficient Off-Policy Distillation via Variance-Guided Reuse

arXiv AI
Jun 9

Trajectory-Refined Distillation

arXiv:2606. 08432v1 Announce Type: new Abstract: On-policy distillation (OPD) has become a central post-training tool for large language models (LLMs), providing dense per-token teacher supervision along the student's own rollouts.

By Li Jiang, Haoran Xu, Yichuan Ding, Amy Zhang
arXiv AI
Aug 18

Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuning

arXiv:2608. 16333v1 Announce Type: cross Abstract: On-policy distillation (OPD) aligns a student model with a teacher's logit distribution on student-generated trajectories.

By Changhui Sun, Lanbo Liu, Hang Lei, Tong Ling, Jiahang Xie, Zhiyong Zheng, Yujia Wang, Hao Liu, Feng Xiao, Lu Liu, Yanlong Du, Zifeng Cheng, Ziwei Jiang, Qing Gu
arXiv Machine Learning
Nov 6

Learning Dynamics of On-Policy Distillation

The paper investigates the learning dynamics of on‑policy distillation (OPD), a method that combines teacher supervision with training on student‑generated trajectories. It identifies two stages of the feedback loop: converting the distillation signal into parameter updates, and broadcasting those updates to alter predictions and reshape the rollout distribution. The study explains why students may be unresponsive to teacher disagreement and how updates that improve local agreement can lead to degeneration by increasing the likelihood of poor continuations. It also shows that offline distillation can prepare a student for OPD without boosting immediate accuracy, yet OPD can still yield significant accuracy gains after sufficient offline preparation.

By Guan Zhe Hong, Sumeet Ramesh Motwani, Liang Chen, Yangchen Pan, Junchi Yu, Jialin Yu, Philip Torr, Yi Ren
arXiv Machine Learning
Jul 7

Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe

arXiv:2605. 03677v2 Announce Type: replace Abstract: On-policy distillation (OPD) has recently emerged as an effective post-training paradigm for consolidating the capabilities of specialized expert models into a single student model.

By Wenjin Hou, Shangpin Peng, Weinong Wang, Zheng Ruan, Yue Zhang, Zhenglin Zhou, Mingqi Gao, Yifei Chen, Kaiqi Wang, Hongming Yang, Chengquan Zhang, Zhuotao Tian, Han Hu, Yi Yang, Fei Wu, Hehe Fan
arXiv AI
Oct 16

Does On-Policy Distillation Need to Complete Every Rollout?

The paper introduces Stochastic Truncated On-Policy Distillation (STOPD), a method that speeds up on-policy distillation by randomly truncating student rollouts and correcting sampling bias with blockwise inverse-probability weighting. STOPD retains teacher feedback for every generated token and preserves the expected gradient of vanilla OPD while providing theoretical error bounds for common optimizers. Experiments show that STOPD cuts generated tokens by 25–48% and achieves 1.3–2.2× faster training without sacrificing accuracy.

By Zhengxin Zhang, Maosen Tang, Ning Wang, Changhao Li, Jingren Zhou, Bo Dai, Sainyam Galhotra, Alex Townsend, Claire Cardie
arXiv AI
Oct 28

Proposing Better Rollouts for On-Policy Distillation

The paper proposes Mixture Proposal On-Policy Distillation (MPOPD) to address a mismatch issue in on‑policy distillation (OPD) where the rollout policy is fixed to the current student. By treating the rollout policy as an optimization target, MPOPD derives a geometric mixture of student and teacher policies and introduces a proposal policy for efficient rollout generation. Experiments on 16 reasoning benchmarks show consistent gains over OPD, notably a 6.91% accuracy improvement on math reasoning with Qwen3-4B-Base and a 41% reduction in generation length.

By Sirui Chen, Yunzhe Qi, Haobo Xu, Jiaru Zou, Mengting Ai, Zihao Li, Jingrui He
arXiv Computation and Language
Oct 28

Scaling Reasoning Efficiently via Relaxed On-Policy Distillation

The paper introduces REOPOLD, a framework that improves on-policy distillation (OPD) for reasoning tasks by addressing the problem of uniform credit allocation. REOPOLD selectively controls where teacher feedback is applied, its strength, and the timing of changes during training, thereby reducing entropy collapse and negative transfer. Experiments show that REOPOLD achieves up to 12× higher sample efficiency compared to recent post‑training baselines and can be extended to cross‑vocabulary distillation, self‑distillation, and test‑time scaling.

By Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, Pashmina Cameron
arXiv AI
Nov 5

KL for a KL: On-Policy Distillation with Control Variate Baseline

The paper introduces vOPD, an On‑Policy Distillation method that incorporates a control variate baseline from reinforcement learning to reduce gradient variance. By using the per‑token negative reverse KL divergence as a closed‑form value function, vOPD maintains a lightweight single‑sample estimator while keeping gradients unbiased. Experiments on mathematics, science, and instruction‑following tasks show vOPD consistently outperforms standard OPD and matches the performance of more expensive full‑vocabulary methods.

By Minjae Oh, Sangjun Song, Gyubin Choi, Yunho Choi, Yohan Jo