arXiv AI

Bootstrapped On-Policy Distillation

Bootstrapped On-Policy Distillation (BOPD) is a two‑stage framework that first uses on‑policy self‑distillation (OPSD) to capture task‑relevant knowledge from reference‑conditioned supervision into an intermediate model, and then distills this intermediate model into a fresh copy of the initial model. The approach addresses the degradation observed in OPSD during continued training, which is caused by a growing mismatch between the evolving student and the frozen teacher. Experiments demonstrate that BOPD improves mathematical reasoning and code generation beyond both the initial models and the constructed OPSD teachers.

arXiv Machine Learning
Aug 20

Rethinking Privileged Information in On-Policy Self-Distillation

The paper investigates on‑policy self‑distillation (OPSD), where a student model learns from its own outputs using token‑level supervision conditioned on privileged reference information. Experiments with Qwen3 models on science and mathematics datasets show that the correct reference does not consistently improve performance; students can improve without it, and solutions from other problems sometimes outperform the correct reference. The study finds that student predictions align more closely with the base model’s reasoning than with the reference supervision, and that alignment alone does not reliably predict performance gains.

By Samyak Shrestha, Alexander Tessier
arXiv Computation and Language
Sep 28

Recursive Self-Improvement via On-Policy Distillation for Reasoning

The paper introduces a recursive self-improvement framework for language models that replaces an external teacher with a frozen copy of the student, enabling dynamic co-evolution (DCE) and self-refined concise learning (SRCL). DCE allows the privileged teacher to evolve alongside the student, while SRCL trains on shorter, verified rewrites to reduce verbosity. Experiments show that the combined DCE+SRCL approach outperforms traditional on‑policy self‑distillation across multiple model sizes and math benchmarks, achieving significant accuracy gains and shorter outputs.

By Shangjian Yin, Zehao Zhao, Kavosh Asadi, Rui Liu, Yuchen Lu, Shike Mei, Hang Cui, Luke Simon, Zhouxing Shi, Hamed Firooz
arXiv Computation and Language
Sep 18

What Does Privileged Information Add to On-Policy Self-Distillation?

The paper investigates how privileged information—such as a teacher’s full solution or reasoning trace—affects on‑policy self‑distillation (OPSD) in language models. Using the AMPLE‑Math benchmark, the authors compare distillation with and without extra teacher views, finding that reference‑free distillation explains most gains for Qwen3‑1.7B, while additional references provide modest benefits, especially for polished solutions. The study also shows that the impact of privileged data depends on the student’s training regime and that altering token‑level supervision can leave student behavior largely unchanged.

By XiuYu Zhang, Wei Chow, Junfeng Fang, Zhenkai Liang, Tat-Seng Chua
arXiv AI
Jul 3

Purified OPSD: On-Policy Self-Distillation Without Losing How to Think

arXiv:2607. 02234v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) has emerged as a promising paradigm for improving LLM reasoning, where a privileged teacher with access to reference solutions provides token-level supervision on the student's own generated trajectories.

By Zhanming Shen, Jintao Tong, Shaotian Yan, Chen Shen, Hao Chen, Wentao Ye, Xiaomeng Hu, Rui Miao, Haobo Wang, Junbo Zhao, Gang Chen, Jieping Ye
arXiv Machine Learning
Aug 27

One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation

The paper reviews On‑Policy Self‑Distillation (OPSD), a method where a language model learns from its own generations using privileged information such as reference solutions or plans, eliminating the need for a larger teacher model. It identifies a key failure mode—collapse, where the model’s reasoning paths narrow progressively—and analyzes it through three levers: signal application, privileged information, and teacher dynamics. The review focuses on mathematical reasoning, offering a unified vocabulary and distinguishing settled facts from ongoing debates.

By Justin Robert, Raheel Qader
arXiv Machine Learning
Oct 22

TeachBack: Learning to Teach via Bidirectional Preference Optimization for On-Policy Self-Distillation

TeachBack is a bidirectional preference-learning framework for on‑policy self‑distillation that teaches a language model not only from a privileged Teacher but also how the Teacher should teach. By controlling the length of the reference‑reasoning suffix and varying prompt style, TeachBack encourages the Teacher to explore and re‑solve problems, reducing reliance on the reference solution while maintaining comparable accuracy. Experiments on mathematical and logical reasoning tasks demonstrate that TeachBack consistently improves Student performance over existing OPSD methods and related baselines.

By Linfei Ji (Huazhong University of Science and Technology), Jiayun Li (Tsinghua University), Haozhao Wang (Huazhong University of Science and Technology), Siyu Jiang (Huazhong University of Science and Technology), Zehao Wang (Harbin Institute of Technology), Longtao Xiao (Huazhong University of Science and Technology), Yu-Jie Yuan (Huawei Technologies Ltd), Ruixuan Li (Huazhong University of Science and Technology)
arXiv Machine Learning
Oct 23

One Teacher, Many Views: Reducing Reference Sensitivity in On-Policy Self-Distillation

The paper introduces Monte Carlo Privileged Distillation (MCPD) to address reference-induced target instability in on-policy self-distillation (OPSD). MCPD models privileged context as a distribution over partial reference views and averages teacher distributions across these views, yielding a target that matches the expected KL divergence minimization. Experiments on mathematical and scientific reasoning benchmarks show that MCPD provides more reference-consistent supervision and consistently outperforms full-reference OPSD.

By Khouloud Saadi, Di Wang
arXiv AI
Aug 11

Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation

arXiv:2608. 08176v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) improves the reasoning abilities of LLMs by internalizing privileged context into model parameters through self-distillation.

By Yongkang Yang, Zhezheng Hao, Hong Zhang, Yi Liu, Xiankun Lin, Wence Ji, Fanjunduo Wei, Jiarui Yu, Qiang Lin, Xiaoyun Liang, Hande Dong