arXiv AI

Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuning

arXiv:2608. 16333v1 Announce Type: cross Abstract: On-policy distillation (OPD) aligns a student model with a teacher's logit distribution on student-generated trajectories.

arXiv AI

Jun 9

Trajectory-Refined Distillation

arXiv:2606. 08432v1 Announce Type: new Abstract: On-policy distillation (OPD) has become a central post-training tool for large language models (LLMs), providing dense per-token teacher supervision along the student's own rollouts.

By Li Jiang, Haoran Xu, Yichuan Ding, Amy Zhang

llms efficiency benchmarks

arXiv AI

Jun 2

Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance

arXiv:2606. 00305v1 Announce Type: cross Abstract: On-Policy Distillation (OPD) improves large language model reasoning by training a student model on trajectories sampled from its own policy under teacher supervision.

By Yuxuan Jiang, Francis Ferraro

llms efficiency

arXiv Machine Learning

Jul 7

Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe

arXiv:2605. 03677v2 Announce Type: replace Abstract: On-policy distillation (OPD) has recently emerged as an effective post-training paradigm for consolidating the capabilities of specialized expert models into a single student model.

By Wenjin Hou, Shangpin Peng, Weinong Wang, Zheng Ruan, Yue Zhang, Zhenglin Zhou, Mingqi Gao, Yifei Chen, Kaiqi Wang, Hongming Yang, Chengquan Zhang, Zhuotao Tian, Han Hu, Yi Yang, Fei Wu, Hehe Fan

llms efficiency multimodal benchmarks

arXiv AI

Jun 16

On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents

arXiv:2606. 15912v1 Announce Type: cross Abstract: Multi-turn agents that plan, invoke tools, and interact with environments offer a promising paradigm for solving complex tasks, yet their capabilities typically rely on very large models whose inference cost is prohibitive in practice.

By Gengsheng Li, Mao Zheng, Mingyang Song, Ruiqi Liu, Tianyu Yang, Jie Sun, Qiyong Zhong, Haiyun Guo, Junfeng Fang, Dan Zhang, Jinqiao Wang

agents efficiency

arXiv Machine Learning

Aug 4

Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rollouts

arXiv:2606. 21994v2 Announce Type: replace Abstract: On-policy distillation (OPD) improves reasoning models by applying dense teacher supervision on student-sampled trajectories.

By Qingfei Zhao, Huan Song, Shuyu Tian, Jiawei Shao, Xuelong Li

efficiency benchmarks

arXiv AI

Jul 7

Multi-Turn On-Policy Distillation with Prefix Replay

arXiv:2607. 04763v1 Announce Type: cross Abstract: We study on-policy distillation (OPD) for agentic tasks, where an LLM agent interacts with an environment over multiple turns and a student imitates a teacher over these multi-turn interaction histories.

By Baohao Liao, Hanze Dong, Christof Monz, Xinxing Xu, Li Dong, Furu Wei

llms agents efficiency

arXiv AI

Jun 24

On the Position Bias of On-Policy Distillation

arXiv:2606. 22600v2 Announce Type: replace-cross Abstract: On-Policy Distillation (OPD) improves the learning efficiency of standard reinforcement learning through dense, token-level supervision from teachers.

By Yan Xie, Sijie Zhu, Tiansheng Wen, Bo Chen, Yifei Wang

reinforcement-learning efficiency safety

arXiv Machine Learning

Aug 4

Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillation

arXiv:2608. 01953v1 Announce Type: cross Abstract: On-policy distillation (OPD) provides teacher supervision on states visited by the student, reducing the distribution gap between training and inference.

By Chishui Chen, Yaoyou Fan, Te Sun, Yi Yang, Chenghao Sun, Delin Mao, Hongbo Qiao, Zuowei Zhang, Junxi Wang, Chenxing Sun, Yangen Hu, Lu Pan, Xuyang Liu, Linfeng Zhang

agents efficiency

arXiv AI

Jul 31

ATOD: Annealed Turn-Aware On-Policy Distillation for Multi-Turn Agentic Tasks

arXiv:2606. 27814v4 Announce Type: replace Abstract: Training small language-model agents for long-horizon interactive tasks requires both fast imitation and reward-driven improvement.

By Qitai Tan, Zefang Zong, Mo Li, Yipeng Shi, Yang Li, Peng Chen

agents reinforcement-learning efficiency

arXiv AI

Jul 7

Reward-Gated On-Policy Distillation

arXiv:2607. 04037v1 Announce Type: cross Abstract: On-policy distillation is a powerful way to transfer reasoning ability from a strong teacher to a smaller student: the student samples trajectories from its own policy, and the teacher provides dense token-level supervision on the states the student actually visits.

By Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi, Chanakya Ekbote, Sujay Sanghavi, Aleksandar Bojchevski

efficiency benchmarks

arXiv Machine Learning

Aug 11

Adaptive Supervised Anchoring for On-Policy Self-Distillation

arXiv:2608. 07935v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) adapts a language model by distilling guidance from a frozen teacher on trajectories sampled from the student.

By Meilin Yang (Renmin University of China, Beijing, China), Zixuan Ding (Renmin University of China, Beijing, China), Jianhao Nie (Renmin University of China, Beijing, China), Weite Zhang (Renmin University of China, Beijing, China), Yuxin Zhang (Renmin University of China, Beijing, China), Zhiming Shao (Renmin University of China, Beijing, China), Li Yu (Renmin University of China, Beijing, China), Zhe Fu (Renmin University of China, Beijing, China)

llms efficiency benchmarks safety

arXiv AI

Jun 29

ATOD: Annealed Turn-aware On-policy Distillation for Multi-turn Autonomous Agents

arXiv:2606. 27814v1 Announce Type: new Abstract: Training small language-model agents for long-horizon interactive tasks requires both fast imitation and reward-driven improvement.

By Qitai Tan, Zefang Zong, Yang Li, Peng Chen

agents reinforcement-learning efficiency

Related stories