Hugging Face Trending Papers
One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
Related stories
arXiv Machine Learning
Aug 27
One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation
The paper reviews On‑Policy Self‑Distillation (OPSD), a method where a language model learns from its own generations using privileged information such as reference solutions or plans, eliminating the need for a larger teacher model. It identifies a key failure mode—collapse, where the model’s reasoning paths narrow progressively—and analyzes it through three levers: signal application, privileged information, and teacher dynamics. The review focuses on mathematical reasoning, offering a unified vocabulary and distinguishing settled facts from ongoing debates.
arXiv AI
Sep 30
Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models
arXiv:2609.37132v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) improves large language models by letting a self-teacher with privileged information provide dense token-level super...
arXiv Machine Learning
Oct 9
Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals
The paper investigates why on‑policy distillation (OPD) can fail, observing that larger‑scale teachers often hit early loss plateaus while self‑RL teachers achieve much greater loss reductions. By modeling OPD as a continuous‑time dynamical system and analyzing training logs, the authors link plateaus to a rapid decline in a gradient‑based learning‑signal proxy, even though loss remains high. They also prove a local recovery guarantee for teachers close to the initial student and note that minimal parameter changes and high representation similarity suggest limited adaptation may cause learning‑signal collapse.
Hugging Face Trending Papers
Oct 20
Projected Distillation: On Learning to Solve Hard Problems from Close, Correct Targets
On-policy learning is central to post-training: training on a model's own generations limits distribution shift and helps preserve existing capabilities. On hard problems, however, on-policy RL provid...
arXiv AI
Aug 6
Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation
arXiv:2608. 04794v1 Announce Type: new Abstract: Self-distillation (SD) has emerged as a compute-efficient alternative to reinforcement learning with verifiable rewards: a self-teacher, conditioned on privileged information (PI) about the answer such as a reference solution, supplies dense per-token supervision to a student that never sees it.
arXiv Computation and Language
Sep 28
Recursive Self-Improvement via On-Policy Distillation for Reasoning
The paper introduces a recursive self-improvement framework for language models that replaces an external teacher with a frozen copy of the student, enabling dynamic co-evolution (DCE) and self-refined concise learning (SRCL). DCE allows the privileged teacher to evolve alongside the student, while SRCL trains on shorter, verified rewrites to reduce verbosity. Experiments show that the combined DCE+SRCL approach outperforms traditional on‑policy self‑distillation across multiple model sizes and math benchmarks, achieving significant accuracy gains and shorter outputs.