arXiv Machine Learning June 10, 2026 By Xinyue Peng, Yi Qian, Jiaojiao Lin, Wenjian Shao, Yanming Liu

PADD: Path-Aligned Decompression Distillation for Non-Router Teacher to Guide MoE Student Learning

Read the original on arXiv Machine Learning →

arXiv:2606. 10369v1 Announce Type: cross Abstract: As large language models (LLMs) continue to scale, it becomes increasingly challenging to grow model capacity under fixed computation budgets.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Jul 24

Student-Centered Distillation Narrows the Agentic Gap Between Small and Large LLMs

arXiv:2509. 14257v3 Announce Type: replace-cross Abstract: Large Language Model agents achieve strong performance on multi-step reasoning and tool-use tasks, but their impressive capabilities typically rely on extremely large backbones.

By Yuanjie Lyu, Chengyu Wang, Jun Huang, Tong Xu

llms agents reinforcement-learning fine-tuning efficiency benchmarks

arXiv AI

Jun 19

Reinforcement-aware Knowledge Distillation for LLM Reasoning

arXiv:2602. 22495v3 Announce Type: replace-cross Abstract: Reinforcement learning (RL) post-training has recently driven major gains in long chain-of-thought reasoning large language models (LLMs), but the high inference cost of such models motivates distillation into smaller students.

By Zhaoyang Zhang, Shuli Jiang, Yantao Shen, Yuting Zhang, Dhananjay Ram, Shuo Yang, Zhuowen Tu, Wei Xia, Stefano Soatto

llms reinforcement-learning fine-tuning efficiency benchmarks

arXiv Machine Learning

Jun 19

A Survey of On-Policy Distillation for Large Language Models

arXiv:2604. 00626v4 Announce Type: replace Abstract: As Large Language Models continue to grow in both capability and cost, transferring frontier capabilities into smaller, deployable students has become an important engineering problem, and knowledge distillation remains a common technique for this transfer.

By Mingyang Song, Mao Zheng

llms agents reinforcement-learning efficiency safety

arXiv AI

Jun 19

Wisdom of Committee: Diverse Distillation from Large Foundation Models and Domain Experts

arXiv:2402. 14035v4 Announce Type: replace-cross Abstract: Knowledge distillation from foundation models to compact domain models is challenging due to substantial gaps in capacity, architecture, and modality.

By Zichang Liu, Qingyun Liu, Yuening Li, Liang Liu, Anshumali Shrivastava, Shuchao Bi, Lichan Hong, Ed H. Chi, Zhe Zhao

llms efficiency

arXiv AI

Aug 21

Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress

arXiv:2608. 19408v1 Announce Type: new Abstract: On-policy distillation (OPD) has emerged as an effective framework for post-training language models by pairing student-generated trajectories with dense token-level supervision from a teacher.

By Chen Yang, Haiyuan Wan, Rengrong Xiong, Yize Chen, Danny H. K. Tsang

llms efficiency

arXiv Machine Learning

4d ago

REVO: Rollout-Efficient Off-Policy Distillation via Variance-Guided Reuse

arXiv:2609.37500v1 Announce Type: new Abstract: On-policy distillation (OPD) trains language models using dense token-level teacher supervision on student-generated trajectories. However, its relianc...

By Yuxiao Yang, Shangzhe Li, Tianrun Yu, Kaixiang Zhao, Taylor W. Killian, Weitong Zhang

llms efficiency benchmarks

One email a morning, machine-written

Related stories

Student-Centered Distillation Narrows the Agentic Gap Between Small and Large LLMs

Reinforcement-aware Knowledge Distillation for LLM Reasoning

A Survey of On-Policy Distillation for Large Language Models

Wisdom of Committee: Diverse Distillation from Large Foundation Models and Domain Experts

Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress

REVO: Rollout-Efficient Off-Policy Distillation via Variance-Guided Reuse