arXiv AI

Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation

arXiv:2608. 09228v1 Announce Type: cross Abstract: On-Policy Self-Distillation (OPSD) is commonly interpreted as the transfer of privileged information: a teacher observes the verified solution to the target problem and supervises the student's trajectory.

arXiv AI

5d ago

DAPD: Dual-Anchored Policy Distillation

arXiv:2608. 01735v2 Announce Type: replace Abstract: On-policy (self) distillation (OPSD) is increasingly adopted for language-model post-training.

By Jianyu Wu, Yizhou Wang, Encheng Su, Chen Tang, Shixiang Tang

efficiency

arXiv AI

Aug 6

Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation

arXiv:2608. 04794v1 Announce Type: new Abstract: Self-distillation (SD) has emerged as a compute-efficient alternative to reinforcement learning with verifiable rewards: a self-teacher, conditioned on privileged information (PI) about the answer such as a reference solution, supplies dense per-token supervision to a student that never sees it.

By Sarthak Harne, Chinmay Karkar, Yash Pandya, Ahmed Awadallah, Akshay Nambi

agents nlp reinforcement-learning efficiency safety

arXiv Machine Learning

Jul 30

Weak-to-Strong On-Policy Distillation

arXiv:2607. 26246v1 Announce Type: new Abstract: On-policy distillation (OPD), which aligns a student with the teacher's token-level distribution on the student's own rollouts, is an effective paradigm for transferring capabilities across LLMs.

By Fangxu Yu, Zinan Lin, Xiaodong Liu, Weijia Xu, Michael Xu, Tianyi Zhou, Jianfeng Gao

llms reinforcement-learning efficiency benchmarks

arXiv AI

Aug 11

SR-OPSD: Self-Referenced On-Policy Self-Distillation

arXiv:2608. 09745v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD) converts feedback into dense token-level supervision on trajectories generated by the policy to be optimized, providing a useful complement to reinforcement learning with sparse outcome rewards.

By Zhuo Sun, Entong Li, Yanlong Zhao, Xiaoyuan Cheng, Wenxuan Yuan, Kaiyu Li, Che Liu, Huihang Liu, Harrison Bo Hua Zhu, Li Zeng

llms reinforcement-learning efficiency benchmarks

arXiv AI

Aug 11

PAST: Privileged Adaptation from Complete Student Trajectories for On-Policy Self-Distillation

arXiv:2608. 08726v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD) uses a privileged teacher to supervise a reasoning model on prefixes sampled from its own rollouts.

By Yangyang Feng, Zhuoyan Feng, Junlan Chen

efficiency benchmarks

arXiv AI

Jun 10

Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning

arXiv:2605. 11458v3 Announce Type: replace Abstract: On-policy self-distillation has become a strong recipe for LLM reasoning, where a privileged teacher supervises the student's own rollouts while conditioning on the reference solution.

By Zihao Han, Tiangang Zhang, Huaibin Wang, Yilun Sun

llms reinforcement-learning efficiency

arXiv AI

Aug 11

Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillation

arXiv:2608. 09826v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards yields no group-relative signal when rollout groups are uniformly correct or uniformly wrong, which account for 63.

By Yubo Jiang, Fengying Xie, Zhiguo Jiang, Haopeng Zhang

reinforcement-learning efficiency benchmarks

arXiv AI

Aug 11

Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation

arXiv:2608. 08176v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) improves the reasoning abilities of LLMs by internalizing privileged context into model parameters through self-distillation.

By Yongkang Yang, Zhezheng Hao, Hong Zhang, Yi Liu, Xiankun Lin, Wence Ji, Fanjunduo Wei, Jiarui Yu, Qiang Lin, Xiaoyun Liang, Hande Dong

llms efficiency benchmarks

Hugging Face Trending Papers

Aug 10

Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillation

Reinforcement learning with verifiable rewards yields no group-relative signal when rollout groups are uniformly correct or uniformly wrong, which account for 63. 0-68.

reinforcement-learning efficiency benchmarks

arXiv AI

Jun 30

DOPD: Dual On-policy Distillation

arXiv:2606. 30626v1 Announce Type: new Abstract: On-policy distillation (OPD) offers superior capacity transfer by supervising student-sampled trajectories with dense token-level signals.

By Xinlei Yu, Gen Li, Qingyi Si, Guibin Zhang, Yuqi Xu, Congcong Wang, Shuai Dong, Kaiwen Tuo, Xiangyu Zeng, Kaituo Feng, Qunzhong Wang, Yang Shi, Xiaobin Hu, Xiangyu Yue, Jiaqi Wang, Shuicheng Yan

llms efficiency multimodal

arXiv AI

Jul 3

DemoPSD: Disagreement-Modulated Policy Self-Distillation

arXiv:2607. 02502v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access.

By Yunhe Li, Hao Shi, Wenhao Liu, Mengzhe Ruan, Hanxu Hou, Zhongxiang Dai, Shuang Qiu, Linqi Song

llms efficiency benchmarks

arXiv AI

Jul 7

Rethinking On-Policy Self-Distillation for Thinking Models

arXiv:2607. 05184v1 Announce Type: new Abstract: Self-distillation is a promising recipe for self-improvement in language models.

By Simran Kaur, Narutatsu Ri, Yinghui He, Liam Fowl, Sanjeev Arora

llms efficiency

Related stories