Hugging Face Trending Papers

Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models

arXiv AI
Jul 7

Reward-Gated On-Policy Distillation

arXiv:2607. 04037v1 Announce Type: cross Abstract: On-policy distillation is a powerful way to transfer reasoning ability from a strong teacher to a smaller student: the student samples trajectories from its own policy, and the teacher provides dense token-level supervision on the states the student actually visits.

By Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi, Chanakya Ekbote, Sujay Sanghavi, Aleksandar Bojchevski
arXiv Machine Learning
Nov 6

Reinforcing What the Student Has Learned: Teacher-Guided On-Policy Distillation

The paper introduces Student-Delta On-Policy Distillation (SD-OPD), a method that refines language-model reasoning by using the student’s own pretrained base as a reference for teacher feedback. Unlike traditional on‑policy distillation (OPD), SD‑OPD scales updates based on centered student‑base differences, applies them only when they align with teacher‑student feedback, and recomputes the student signal each training round. Experiments with Qwen3 students ranging from 1.7B to 8B parameters show that SD‑OPD boosts six‑task mathematics average pass@1 by 7.76 to 10.37 percentage points over OPD.

By Bozhen Fang, Linhao Zhong, Xuehao Wang, Huanyi Zheng, Hao Chen, Chunhua Shen
arXiv AI
Aug 11

Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation

arXiv:2608. 08176v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) improves the reasoning abilities of LLMs by internalizing privileged context into model parameters through self-distillation.

By Yongkang Yang, Zhezheng Hao, Hong Zhang, Yi Liu, Xiankun Lin, Wence Ji, Fanjunduo Wei, Jiarui Yu, Qiang Lin, Xiaoyun Liang, Hande Dong
arXiv AI
Oct 9

MetaOPD: Meta-Learned Token Weighting for On-Policy Distillation

MetaOPD introduces a bilevel optimization framework that jointly learns a student model and a lightweight token‑weighting network for on‑policy distillation. The inner objective updates the student using weighted OPD, while the outer objective optimizes the weighting network based on validation loss after a virtual student update, enabling the token‑weight mapping to adapt to the student’s evolving needs. Experiments on multiple reasoning and out‑of‑domain datasets show that MetaOPD improves performance over standard OPD by up to 6.41 percentage points on larger student models.

By Zipeng Wang, Xinpeng Dong, Yuefan Wang, Pingchen Lu, Xian Wei, Kun Kuang, Fei Wu, Zhongxiang Dai, Min Zhang
arXiv AI
Oct 28

Proposing Better Rollouts for On-Policy Distillation

The paper proposes Mixture Proposal On-Policy Distillation (MPOPD) to address a mismatch issue in on‑policy distillation (OPD) where the rollout policy is fixed to the current student. By treating the rollout policy as an optimization target, MPOPD derives a geometric mixture of student and teacher policies and introduces a proposal policy for efficient rollout generation. Experiments on 16 reasoning benchmarks show consistent gains over OPD, notably a 6.91% accuracy improvement on math reasoning with Qwen3-4B-Base and a 41% reduction in generation length.

By Sirui Chen, Yunzhe Qi, Haobo Xu, Jiaru Zou, Mengting Ai, Zihao Li, Jingrui He
arXiv Machine Learning
Sep 1

Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement

The paper investigates on‑policy distillation (OPD), showing that teacher supervision during OPD contains significant noise that grows with teacher size, yet the student policy remains largely unaffected by this noise. It finds that OPD’s gains stem mainly from suppressing low‑log‑probability tokens, a process that can be replicated without a teacher. Building on this insight, the authors propose On‑Policy Self‑Adaptation (OPSA), a supervision‑free method that uses entropy‑adaptive negative advantages to improve performance on several benchmarks, outperforming both the base model and OPD.

By Yi Ding, Ruqi Zhang
arXiv Machine Learning
Jul 22

H$^2$SD: Hybrid Hindsight Self-Distillation

arXiv:2607. 18955v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models on tasks such as mathematical reasoning and code generation.

By Qiye Cai, Yichuan Ma, Linyang Li, Peiji Li, Yongkang Chen, Qipeng Guo, Yicheng Zou, Tao Gui, Xiaocheng Feng, Bing Qin