Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
The Flow has not summarised this story yet — read it at arXiv AI.
Bootstrapped On-Policy Distillation (BOPD) is a two‑stage framework that first uses on‑policy self‑distillation (OPSD) to capture task‑relevant knowledge from reference‑conditioned supervision into an intermediate model, and then distills this intermediate model into a fresh copy of the initial model. The approach addresses the degradation observed in OPSD during continued training, which is caused by a growing mismatch between the evolving student and the frozen teacher. Experiments demonstrate that BOPD improves mathematical reasoning and code generation beyond both the initial models and the constructed OPSD teachers.
The paper introduces a recursive self-improvement framework for language models that replaces an external teacher with a frozen copy of the student, enabling dynamic co-evolution (DCE) and self-refined concise learning (SRCL). DCE allows the privileged teacher to evolve alongside the student, while SRCL trains on shorter, verified rewrites to reduce verbosity. Experiments show that the combined DCE+SRCL approach outperforms traditional on‑policy self‑distillation across multiple model sizes and math benchmarks, achieving significant accuracy gains and shorter outputs.
The paper introduces temporal self‑distillation for reinforcement learning with verifiable rewards (RLVR), proposing that a policy can learn from a stronger future checkpoint of itself. Two methods—Near‑Future Policy Optimization (NPO) and Near‑Future Policy Distillation (NPD)—use verified future‑self trajectories and token‑level transfer, respectively, while AutoNPO adaptively selects the optimal future checkpoint. Experiments on eight image‑text benchmarks show that near‑future teachers yield higher performance than far‑future ones, indicating that the balance between new capability and learner compatibility is key.
RISE (Recursive Improvement via Self-Extrapolating Policy Distillation) is a new method that builds a synthetic teacher from a language model’s own RLVR training trajectory. By extrapolating the displacement between the current checkpoint and a trailing anchor in parameter or logit space, RISE transforms sparse outcome-based updates into dense token-level targets without external models or privileged conditioning. The approach recursively refines the student model, combining RLVR and on‑policy distillation, and demonstrates superior performance across mathematical reasoning, STEM, code generation, and multi‑turn agentic tasks.
arXiv:2607. 18955v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models on tasks such as mathematical reasoning and code generation.
TeachBack is a bidirectional preference-learning framework for on‑policy self‑distillation that teaches a language model not only from a privileged Teacher but also how the Teacher should teach. By controlling the length of the reference‑reasoning suffix and varying prompt style, TeachBack encourages the Teacher to explore and re‑solve problems, reducing reliance on the reference solution while maintaining comparable accuracy. Experiments on mathematical and logical reasoning tasks demonstrate that TeachBack consistently improves Student performance over existing OPSD methods and related baselines.