arXiv AI
PreSO: Privileged Representation Isolation for On-Policy Self-Distillation
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
Sep 28
Recursive Self-Improvement via On-Policy Distillation for Reasoning
The paper introduces a recursive self-improvement framework for language models that replaces an external teacher with a frozen copy of the student, enabling dynamic co-evolution (DCE) and self-refined concise learning (SRCL). DCE allows the privileged teacher to evolve alongside the student, while SRCL trains on shorter, verified rewrites to reduce verbosity. Experiments show that the combined DCE+SRCL approach outperforms traditional on‑policy self‑distillation across multiple model sizes and math benchmarks, achieving significant accuracy gains and shorter outputs.
arXiv Machine Learning
Jul 22
H$^2$SD: Hybrid Hindsight Self-Distillation
arXiv:2607. 18955v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models on tasks such as mathematical reasoning and code generation.
arXiv Machine Learning
Oct 8
A Good Self-Teacher Meets the Student Where They Are: Joint On-Policy Learning and Teaching
The paper introduces Joint On-Policy Learning and Teaching (JOLT), a method that trains a single policy to act both as a privileged teacher and an unprivileged student. It derives a condition ensuring the teacher’s distillation update aligns positively with the student’s reward gradient, leading to a KL‑regularized teacher objective that matches the student’s current capabilities. Experiments show JOLT improves training efficiency and performance across various tasks, with additional gains when student rewards are incorporated.
arXiv AI
Sep 25
Not Every Token Is Worth Distilling: Selective Supervision for Direct-OPD
The paper introduces Selective Supervision for Direct-OPD (S$^2$D-OPD), a refinement of Direct On-Policy Distillation that filters out states where the teacher’s policy change is minimal, as measured by the teacher‑reference Jensen‑Shannon divergence. By masking low‑divergence states and keeping only the top 10% of states per response, S$^2$D-OPD improves held‑out accuracy on AIME and HMMT benchmarks across multiple teacher‑student pairs without additional forward passes.
arXiv AI
Sep 7
RISE: Recursive Improvement via Self-Extrapolating Policy Distillation
RISE (Recursive Improvement via Self-Extrapolating Policy Distillation) is a new method that builds a synthetic teacher from a language model’s own RLVR training trajectory. By extrapolating the displacement between the current checkpoint and a trailing anchor in parameter or logit space, RISE transforms sparse outcome-based updates into dense token-level targets without external models or privileged conditioning. The approach recursively refines the student model, combining RLVR and on‑policy distillation, and demonstrates superior performance across mathematical reasoning, STEM, code generation, and multi‑turn agentic tasks.
arXiv AI
Aug 11
SR-OPSD: Self-Referenced On-Policy Self-Distillation
arXiv:2608. 09745v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD) converts feedback into dense token-level supervision on trajectories generated by the policy to be optimized, providing a useful complement to reinforcement learning with sparse outcome rewards.