Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals

The paper investigates why on‑policy distillation (OPD) can fail, observing that larger‑scale teachers often hit early loss plateaus while self‑RL teachers achieve much greater loss reductions. By modeling OPD as a continuous‑time dynamical system and analyzing training logs, the authors link plateaus to a rapid decline in a gradient‑based learning‑signal proxy, even though loss remains high. They also prove a local recovery guarantee for teachers close to the initial student and note that minimal parameter changes and high representation similarity suggest limited adaptation may cause learning‑signal collapse.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

• llms
• reinforcement-learning
• efficiency