arXiv Machine Learning
What Should a Self-Teacher See? Privileged Context Design for On-Policy Self-Distillation
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Aug 11
Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation
arXiv:2608. 09228v1 Announce Type: cross Abstract: On-Policy Self-Distillation (OPSD) is commonly interpreted as the transfer of privileged information: a teacher observes the verified solution to the target problem and supervises the student's trajectory.
arXiv Machine Learning
1d ago
Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning
arXiv:2609.37915v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) trains a student to match a privileged teacher distribution along its own sampled trajectory. Standard OPSD applies...
arXiv Computation and Language
Sep 18
What Does Privileged Information Add to On-Policy Self-Distillation?
The paper investigates how privileged information—such as a teacher’s full solution or reasoning trace—affects on‑policy self‑distillation (OPSD) in language models. Using the AMPLE‑Math benchmark, the authors compare distillation with and without extra teacher views, finding that reference‑free distillation explains most gains for Qwen3‑1.7B, while additional references provide modest benefits, especially for polished solutions. The study also shows that the impact of privileged data depends on the student’s training regime and that altering token‑level supervision can leave student behavior largely unchanged.
arXiv AI
Aug 6
Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation
arXiv:2608. 04794v1 Announce Type: new Abstract: Self-distillation (SD) has emerged as a compute-efficient alternative to reinforcement learning with verifiable rewards: a self-teacher, conditioned on privileged information (PI) about the answer such as a reference solution, supplies dense per-token supervision to a student that never sees it.
arXiv AI
Jul 7
Rethinking On-Policy Self-Distillation for Thinking Models
arXiv:2607. 05184v1 Announce Type: new Abstract: Self-distillation is a promising recipe for self-improvement in language models.
arXiv AI
1d ago
Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models
arXiv:2609.37132v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) improves large language models by letting a self-teacher with privileged information provide dense token-level super...