arXiv Machine Learning June 15, 2026 By Guo Yu, Wenlin Liu, Yulan Hu, Hao-Xuan Ma, Jun-Peng Jiang, Han-Jia Ye

Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation

Read the original on arXiv Machine Learning →

arXiv:2606. 13657v2 Announce Type: replace Abstract: On-policy distillation (\textsc{OPD}) has recently become a prominent post-training recipe by combining two desirable ingredients: on-policy student trajectories and dense teacher supervision.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

4d ago

Understanding LLM Parameter Update Sparsity through the Lens of Fisher

arXiv:2609.36262v1 Announce Type: new Abstract: Recent studies have observed that parameter changes during language-model post-training can be concentrated in a small subset of coordinates. This phen...

By Yufan Zhang, Sagnik Mukherjee, Hao Peng

llms reinforcement-learning fine-tuning efficiency

arXiv AI

Aug 5

When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation

arXiv:2608. 03632v1 Announce Type: new Abstract: On-Policy distillation (OPD) transfers teacher capabilities by supervising student-sampled trajectories with dense token-level teacher signals.

By Yinuo Jiang, Yongjie Ye, Zhou Tao, Xiang Zhuang, Qiang Zhang, Huajun Chen, Tiankai Li

llms efficiency multimodal

arXiv AI

Jun 8

On the Geometry of On-Policy Distillation

arXiv:2606. 07082v1 Announce Type: cross Abstract: On-policy distillation (OPD) is increasingly used to improve large language model reasoning, but its training dynamics remain poorly understood.

By Zhennan Shen, Yanshu Li, Qingyu Yin, Chak Tou Leong, Zhilin Wang, Yanxu Chen, Rongduo Han, Sunbowen Lee, Yi R. Fung

llms reinforcement-learning fine-tuning efficiency

arXiv AI

Sep 25

Not Every Token Is Worth Distilling: Selective Supervision for Direct-OPD

The paper introduces Selective Supervision for Direct-OPD (S$^2$D-OPD), a refinement of Direct On-Policy Distillation that filters out states where the teacher’s policy change is minimal, as measured by the teacher‑reference Jensen‑Shannon divergence. By masking low‑divergence states and keeping only the top 10% of states per response, S$^2$D-OPD improves held‑out accuracy on AIME and HMMT benchmarks across multiple teacher‑student pairs without additional forward passes.

By Yibo Zhao, Zixuan Yang, Yunshi Lan, Xiang Li

reinforcement-learning efficiency benchmarks

arXiv AI

Aug 11

SR-OPSD: Self-Referenced On-Policy Self-Distillation

arXiv:2608. 09745v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD) converts feedback into dense token-level supervision on trajectories generated by the policy to be optimized, providing a useful complement to reinforcement learning with sparse outcome rewards.

By Zhuo Sun, Entong Li, Yanlong Zhao, Xiaoyuan Cheng, Wenxuan Yuan, Kaiyu Li, Che Liu, Huihang Liu, Harrison Bo Hua Zhu, Li Zeng

llms reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Jul 28

FlowCTS: On-policy Continuous Trajectory Supervision of Flow Models

arXiv:2607. 24522v1 Announce Type: new Abstract: While on-policy distillation (OPD) effectively addresses sparse rewards and exposure bias in large language model post-training, its extension to flow models remains underexplored.

By Kaiyang Ye, Yuan Ge, Junxiang Zhang, Bei Li, Ziming Zhu, Haishu Zhao, Xiaoqian Liu, Chenglong Wang, Jingbo Zhu, Zhengtao Yu, Tong Xiao

llms computer-vision reinforcement-learning fine-tuning efficiency safety

One email a morning, machine-written

Related stories

Understanding LLM Parameter Update Sparsity through the Lens of Fisher

When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation

On the Geometry of On-Policy Distillation

Not Every Token Is Worth Distilling: Selective Supervision for Direct-OPD

SR-OPSD: Self-Referenced On-Policy Self-Distillation

FlowCTS: On-policy Continuous Trajectory Supervision of Flow Models