arXiv Machine Learning

Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning

arXiv Computation and Language
Sep 18

What Does Privileged Information Add to On-Policy Self-Distillation?

The paper investigates how privileged information—such as a teacher’s full solution or reasoning trace—affects on‑policy self‑distillation (OPSD) in language models. Using the AMPLE‑Math benchmark, the authors compare distillation with and without extra teacher views, finding that reference‑free distillation explains most gains for Qwen3‑1.7B, while additional references provide modest benefits, especially for polished solutions. The study also shows that the impact of privileged data depends on the student’s training regime and that altering token‑level supervision can leave student behavior largely unchanged.

By XiuYu Zhang, Wei Chow, Junfeng Fang, Zhenkai Liang, Tat-Seng Chua
arXiv Machine Learning
Aug 27

One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation

The paper reviews On‑Policy Self‑Distillation (OPSD), a method where a language model learns from its own generations using privileged information such as reference solutions or plans, eliminating the need for a larger teacher model. It identifies a key failure mode—collapse, where the model’s reasoning paths narrow progressively—and analyzes it through three levers: signal application, privileged information, and teacher dynamics. The review focuses on mathematical reasoning, offering a unified vocabulary and distinguishing settled facts from ongoing debates.

By Justin Robert, Raheel Qader
arXiv Computation and Language
Sep 28

Recursive Self-Improvement via On-Policy Distillation for Reasoning

The paper introduces a recursive self-improvement framework for language models that replaces an external teacher with a frozen copy of the student, enabling dynamic co-evolution (DCE) and self-refined concise learning (SRCL). DCE allows the privileged teacher to evolve alongside the student, while SRCL trains on shorter, verified rewrites to reduce verbosity. Experiments show that the combined DCE+SRCL approach outperforms traditional on‑policy self‑distillation across multiple model sizes and math benchmarks, achieving significant accuracy gains and shorter outputs.

By Shangjian Yin, Zehao Zhao, Kavosh Asadi, Rui Liu, Yuchen Lu, Shike Mei, Hang Cui, Luke Simon, Zhouxing Shi, Hamed Firooz
arXiv Computation and Language
Oct 28

Bridged Self-Distillation for On-Policy Learning

Bridged Self-Distillation (BSD) is a unified framework that enables on‑policy distillation from any teacher providing reasoning traces. The external teacher supplies a privileged reasoning trace offline, while a self‑teacher—an identical copy of the student conditioned on that trace—scores the student’s rollouts in its own token space. BSD uses these token‑level signals, along with a verifier, to adjust updates on unsuccessful rollouts, allowing the same training procedure to work with shared‑tokenizer, cross‑tokenizer, and closed‑API teachers and consistently outperforming scenario‑specific baselines.

By Runsheng Chen, Yihong Liu, Raoyuan Zhao, Michael A. Hedderich, Hinrich Sch\"utze, Matthias Grabmair
arXiv AI
Oct 7

E$^2$-OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation

E$^2$-OPSD tackles a failure mode in on‑policy self‑distillation where the student’s token entropy surpasses the teacher’s, a problem called entropy overshoot. The method replaces the teacher’s answer with a retrieved solved neighboring problem to provide transferable reasoning guidance and uses the student‑teacher entropy gap to adjust token corrections. These changes improve math reasoning scores by up to 4.3 points in mean@16 and yield out‑of‑domain gains of up to 4.9 points in mean@16 and 5.5 points in pass@8, all without extra forward passes or networks.

By Yifei Liu, Minghao Fang, Xinyu Gu, Chengkai Yao, Mengdi Liu, Tengfei Ma, Jiangbin Zheng, Chang Yu, Zhangyang Gao
arXiv AI
Sep 3

Learn from Whoever Is Right: Answer-Verified Multi-Teacher Distillation for Multi-Domain LLMs

The paper introduces Multi-Teacher Self-Distillation Policy Optimization (MT‑SDPO), an on‑policy distillation method that combines multiple frozen teachers into a single student model. MT‑SDPO uses self‑anchors, answer‑verified eligibility, and privileged distillation to select reliable teachers per sample rather than per domain. Experiments on five students from three model families show that MT‑SDPO improves the weakest domain of Qwen3‑8B by 14.79 points and reduces its domain gap by 74.7%, achieving a more balanced performance than matching a single teacher to each domain.

By Xixiang He, Xingming Li, Baiqi Wu, Qiyao Sun, Xuanyu Ji, Ao Cheng, Qingyong Hu