← Back to all news
arXiv AI By Bangji Yang, Jiajun Fan, Zhaoyu Zhu, Hongbo Ma, Ge Liu

Training-Free On-Policy Distillation

Read the original on arXiv AI →

The paper introduces training‑free on‑policy distillation, a method that consolidates multiple specialist models into a single student without any gradient updates. By performing only forward passes over a few hundred rollouts, the fused model can reproduce each specialist’s outputs layer‑by‑layer, achieving higher average accuracy than any individual specialist. The approach relies on a least‑squares solution that uses only the second moments of inputs, and careful handling of rollouts—such as removing repetitive answers—significantly improves performance on benchmarks like AIME25 and Olmo 3‑7B.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Oct 28

Proposing Better Rollouts for On-Policy Distillation

The paper proposes Mixture Proposal On-Policy Distillation (MPOPD) to address a mismatch issue in on‑policy distillation (OPD) where the rollout policy is fixed to the current student. By treating the rollout policy as an optimization target, MPOPD derives a geometric mixture of student and teacher policies and introduces a proposal policy for efficient rollout generation. Experiments on 16 reasoning benchmarks show consistent gains over OPD, notably a 6.91% accuracy improvement on math reasoning with Qwen3-4B-Base and a 41% reduction in generation length.

By Sirui Chen, Yunzhe Qi, Haobo Xu, Jiaru Zou, Mengting Ai, Zihao Li, Jingrui He
efficiencybenchmarks
More like this →
arXiv AI
Oct 27

DoubleFree:Exploiting Exploration History for Rollout-Free Multi-Task LLMs Training

DoubleFree is a method that reuses exploration trajectories from multi-task reinforcement learning to enable rollout‑free consolidation of large language models. By precomputing teacher supervision from historical data, it eliminates the need for online teachers during distillation. Experiments on Qwen3 and SmolLM models show speedups of 3–6× and GPU‑hour reductions of 81–90% while preserving performance across math, code, and instruction tasks.

By Yingping Wang, Min Cai, Chenyu Yuan, Xi Ye, Zhijiang Guo, Jiayi Huang
llmsreinforcement-learningefficiency
More like this →
arXiv AI
Oct 15

LoOPSD: On-Policy Self-Distillation with Looped Teachers

arXiv:2610.18162v1 Announce Type: cross Abstract: On-policy self-distillation improves a model using guidance from the model itself, but it still requires a teacher that provides more useful predicti...

By Wei Fan, Qingyue Yang, Chao-Hong Tan, Qian Chen, Wen Wang, Xiangang Li, Kejiang Chen, Weiming Zhang, Nenghai Yu
efficiency
More like this →