Mitigating the Length-Scaling Tax with Online Distillation

The paper introduces Length Self-Distillation (LSD) to address the length‑scaling tax (LST) that occurs during reinforcement‑learning post‑training, where models produce unnecessarily verbose responses to already‑solved prompts. LSD routes solved prompts to an on‑policy distillation process while keeping the original RL objective for unsolved prompts, using an exponential moving average of the online policy as its teacher. Experiments show LSD matches or surpasses RL performance while reducing LST from 19.0% to –3.7% on single‑turn reasoning and from 31.4% to 13.7% on multi‑turn agentic tasks, thereby maintaining concise responses on easy queries while still enabling exploration on difficult ones.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• agents
• reinforcement-learning
• efficiency