arXiv AI
LoOPSD: On-Policy Self-Distillation with Looped Teachers
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
Sep 28
Recursive Self-Improvement via On-Policy Distillation for Reasoning
The paper introduces a recursive self-improvement framework for language models that replaces an external teacher with a frozen copy of the student, enabling dynamic co-evolution (DCE) and self-refined concise learning (SRCL). DCE allows the privileged teacher to evolve alongside the student, while SRCL trains on shorter, verified rewrites to reduce verbosity. Experiments show that the combined DCE+SRCL approach outperforms traditional on‑policy self‑distillation across multiple model sizes and math benchmarks, achieving significant accuracy gains and shorter outputs.
arXiv Machine Learning
Oct 9
Recurrent Self-Improvement: Dynamic Cross-Loop On-Policy Distillation for Looped Language Models
LoopOPD introduces a cross-loop on-policy distillation framework that uses a frozen terminal loop policy as a compute-privileged teacher for intermediate loop students, providing dense supervision without external teachers. Dynamic LoopOPD (D-LoopOPD) further refreshes the terminal loop teacher as model parameters update, enabling recurrent self‑improvement. Experiments on Ouro‑Thinking models show that LoopOPD enhances mathematical reasoning and that D‑LoopOPD yields additional gains, with the models also improving on general reasoning and code generation benchmarks.
arXiv AI
Oct 1
STEPS: Selective On-Policy Self-Distillation for Reasoning
arXiv:2605.10194v2 Announce Type: replace Abstract: On-policy self-distillation (OPSD) uses a model as its own teacher under privileged context, providing token-level supervision on the model's own r...
arXiv AI
Oct 2
Masked Self-Distillation: Internalizing the Chain-of-Thought in Language Models
The paper introduces masked self‑distillation, a post‑training framework that trains a language model to internalize portions of its own intermediate reasoning traces. By varying the fraction of trace internalized, the authors demonstrate that models can achieve higher inference efficiency and improved task performance on math and graph‑coloring problems. Experiments on Qwen3‑4B and Qwen3‑8B show that the method generalizes well to in‑domain out‑of‑distribution cases without catastrophic forgetting, and that supervised fine‑tuning alone can reduce trace length at the expense of generalization.
arXiv Machine Learning
Jul 17
On-Policy Delta Distillation
arXiv:2607. 15161v1 Announce Type: new Abstract: On-policy distillation is an alternative post-training method in reinforcement learning that alleviates the constraints imposed by reward models by providing token-level supervision from a teacher model.
arXiv AI
Aug 26
OPDSearch+: On-Policy Distillation with RL Refinement for Search-Augmented Reasoning
OPDSearch+ introduces a two‑stage distillation framework for search‑augmented reasoning that eliminates the need for task‑specific teacher fine‑tuning. In the first stage, a frozen off‑the‑shelf instruct model guides a student through live search interactions using a per‑position forward KL objective, transferring reasoning decomposition and evidence integration skills. The second stage refines this student with reinforcement learning, achieving performance surpassing RL alone and outperforming all prior 3B‑parameter baselines on seven QA benchmarks, including 13.1% improvement on HotpotQA and 8.5% on 2WikiMultihopQA.