arXiv Machine Learning

Adaptive FastOPD: Progress-Aware Rollout Horizon Expansion for Efficient On-Policy Distillation

arXiv:2607. 29494v1 Announce Type: new Abstract: On-policy distillation (OPD) provides dense teacher supervision along student-generated trajectories, but its online rollout process incurs substantial computational cost, particularly when a few long responses delay batch completion.

arXiv AI

1d ago

Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuning

arXiv:2608. 16333v1 Announce Type: cross Abstract: On-policy distillation (OPD) aligns a student model with a teacher's logit distribution on student-generated trajectories.

By Changhui Sun, Lanbo Liu, Hang Lei, Tong Ling, Jiahang Xie, Zhiyong Zheng, Yujia Wang, Hao Liu, Feng Xiao, Lu Liu, Yanlong Du, Zifeng Cheng, Ziwei Jiang, Qing Gu

agents fine-tuning efficiency

arXiv Machine Learning

Aug 4

Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rollouts

arXiv:2606. 21994v2 Announce Type: replace Abstract: On-policy distillation (OPD) improves reasoning models by applying dense teacher supervision on student-sampled trajectories.

By Qingfei Zhao, Huan Song, Shuyu Tian, Jiawei Shao, Xuelong Li

efficiency benchmarks

arXiv AI

Jun 29

ATOD: Annealed Turn-aware On-policy Distillation for Multi-turn Autonomous Agents

arXiv:2606. 27814v1 Announce Type: new Abstract: Training small language-model agents for long-horizon interactive tasks requires both fast imitation and reward-driven improvement.

By Qitai Tan, Zefang Zong, Yang Li, Peng Chen

agents reinforcement-learning efficiency

arXiv AI

Jul 8

TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training

arXiv:2607. 05804v1 Announce Type: new Abstract: On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student's own trajectories, offering a promising framework for language agent training.

By Yuhang Zhou, Kai Zheng, Haoling Li, Dengyun Peng, Can Xu, Jingjing Chen

agents efficiency

arXiv Machine Learning

6d ago

Distillation of Foundation Models for Time-dependent PDEs

arXiv:2608. 11937v1 Announce Type: new Abstract: Foundation models for time-dependent partial differential equations (PDEs) are trained on large and diverse collections of physical systems and can generalize effectively to new downstream tasks.

By Daniel Musekamp, Boshra Ariguib, Andrei Manolache, Mathias Niepert

fine-tuning efficiency benchmarks

arXiv AI

Jul 31

ATOD: Annealed Turn-Aware On-Policy Distillation for Multi-Turn Agentic Tasks

arXiv:2606. 27814v4 Announce Type: replace Abstract: Training small language-model agents for long-horizon interactive tasks requires both fast imitation and reward-driven improvement.

By Qitai Tan, Zefang Zong, Mo Li, Yipeng Shi, Yang Li, Peng Chen

agents reinforcement-learning efficiency

arXiv Machine Learning

Jun 24

AsyncOPD: How Stale Can On-Policy Distillation Be?

arXiv:2606. 24143v1 Announce Type: new Abstract: On-policy distillation (OPD) trains a student on its own rollouts guided by teacher feedback and is becoming increasingly important for large language model (LLM) post-training.

By Wonjun Kang, Kevin Galim, Seunghyuk Oh, Minjun Kang, Sanghyun Park, Donghoon Kim, Minjae Lee, Minseo Kim, Rishabh Tiwari, Yuchen Zeng, Hyung Il Koo, Kangwook Lee

llms reinforcement-learning efficiency safety

arXiv AI

Jun 16

On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents

arXiv:2606. 15912v1 Announce Type: cross Abstract: Multi-turn agents that plan, invoke tools, and interact with environments offer a promising paradigm for solving complex tasks, yet their capabilities typically rely on very large models whose inference cost is prohibitive in practice.

By Gengsheng Li, Mao Zheng, Mingyang Song, Ruiqi Liu, Tianyu Yang, Jie Sun, Qiyong Zhong, Haiyun Guo, Junfeng Fang, Dan Zhang, Jinqiao Wang

agents efficiency

arXiv Machine Learning

Jun 9

Escaping the KL Agreement Trap in On-Policy Distillation

arXiv:2606. 09471v1 Announce Type: new Abstract: On-policy distillation (OPD) provides dense token-level supervision by asking a teacher to score student-generated rollouts.

By Haoran Xin, Anhao Zhao, Ying Sun, Jin Li, Xiaoyu Shen, Hui Xiong

efficiency benchmarks

arXiv Machine Learning

Aug 11

Adaptive Supervised Anchoring for On-Policy Self-Distillation

arXiv:2608. 07935v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) adapts a language model by distilling guidance from a frozen teacher on trajectories sampled from the student.

By Meilin Yang (Renmin University of China, Beijing, China), Zixuan Ding (Renmin University of China, Beijing, China), Jianhao Nie (Renmin University of China, Beijing, China), Weite Zhang (Renmin University of China, Beijing, China), Yuxin Zhang (Renmin University of China, Beijing, China), Zhiming Shao (Renmin University of China, Beijing, China), Li Yu (Renmin University of China, Beijing, China), Zhe Fu (Renmin University of China, Beijing, China)

llms efficiency benchmarks safety

Hugging Face Trending Papers

Jul 27

The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation

Multi-turn long-horizon planning is critical for foundation model agents, yet how to fundamentally improve it remains unclear. Existing models are trained on uncontrollable and opaque Internet data, making it difficult to identify how planning ability is acquired, shaped, and integrated.

agents efficiency

arXiv AI

Jul 28

The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation

arXiv:2607. 24720v1 Announce Type: cross Abstract: Multi-turn long-horizon planning is critical for foundation model agents, yet how to fundamentally improve it remains unclear.

By Tianyi Men, Zhuoran Jin, Kang Liu, Jun Zhao

agents efficiency

Related stories