TeachBack is a bidirectional preference-learning framework for on‑policy self‑distillation that teaches a language model not only from a privileged Teacher but also how the Teacher should teach. By controlling the length of the reference‑reasoning suffix and varying prompt style, TeachBack encourages the Teacher to explore and re‑solve problems, reducing reliance on the reference solution while maintaining comparable accuracy. Experiments on mathematical and logical reasoning tasks demonstrate that TeachBack consistently improves Student performance over existing OPSD methods and related baselines.
arXiv:2609.37132v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) improves large language models by letting a self-teacher with privileged information provide dense token-level super...
The paper introduces Joint On-Policy Learning and Teaching (JOLT), a method that trains a single policy to act both as a privileged teacher and an unprivileged student. It derives a condition ensuring the teacher’s distillation update aligns positively with the student’s reward gradient, leading to a KL‑regularized teacher objective that matches the student’s current capabilities. Experiments show JOLT improves training efficiency and performance across various tasks, with additional gains when student rewards are incorporated.
The paper investigates on‑policy self‑distillation (OPSD), where a student model learns from its own outputs using token‑level supervision conditioned on privileged reference information. Experiments with Qwen3 models on science and mathematics datasets show that the correct reference does not consistently improve performance; students can improve without it, and solutions from other problems sometimes outperform the correct reference. The study finds that student predictions align more closely with the base model’s reasoning than with the reference supervision, and that alignment alone does not reliably predict performance gains.
arXiv:2608. 08176v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) improves the reasoning abilities of LLMs by internalizing privileged context into model parameters through self-distillation.
arXiv:2607. 18955v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models on tasks such as mathematical reasoning and code generation.
The paper introduces Preference‑Based Self‑Distillation (PBSD), a new on‑policy self‑distillation method that replaces traditional KL matching with a reward‑regularized objective. PBSD derives a reward‑reweighted teacher distribution, optimizing preference gaps between teacher and student samples while keeping on‑policy sampling. Experiments on mathematical reasoning and tool‑use tasks show PBSD achieves stronger average performance, improved training stability, and maintains token efficiency compared to prior self‑distillation baselines.
arXiv:2605. 11458v3 Announce Type: replace Abstract: On-policy self-distillation has become a strong recipe for LLM reasoning, where a privileged teacher supervises the student's own rollouts while conditioning on the reference solution.
arXiv:2607. 28582v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) is a promising approach to improve reasoning language models, but it remains brittle in practice: making it work reliably often requires substantial engineering effort.
arXiv:2605. 12400v2 Announce Type: replace-cross Abstract: We study on-policy self-distillation (OPSD), where a language model improves its reasoning ability by distilling privileged teacher distributions along its own on-policy trajectories.
On-policy distillation trains a language model on its own generations while a teacher scores them token by token. It combines the dense supervision of imitation learning with the on-policy sampling of...
The paper introduces a recursive self-improvement framework for language models that replaces an external teacher with a frozen copy of the student, enabling dynamic co-evolution (DCE) and self-refined concise learning (SRCL). DCE allows the privileged teacher to evolve alongside the student, while SRCL trains on shorter, verified rewrites to reduce verbosity. Experiments show that the combined DCE+SRCL approach outperforms traditional on‑policy self‑distillation across multiple model sizes and math benchmarks, achieving significant accuracy gains and shorter outputs.
The paper introduces Monte Carlo Privileged Distillation (MCPD) to address reference-induced target instability in on-policy self-distillation (OPSD). MCPD models privileged context as a distribution over partial reference views and averages teacher distributions across these views, yielding a target that matches the expected KL divergence minimization. Experiments on mathematical and scientific reasoning benchmarks show that MCPD provides more reference-consistent supervision and consistently outperforms full-reference OPSD.