arXiv Machine Learning
EvoOPD: Evolving Privileged Information Strategies for Black-Box On-Policy Distillation
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Oct 19
Predicting Privileged Information Effectiveness for On-Policy Self-Distillation
The paper studies how different configurations of privileged information (PI) affect on‑policy self‑distillation (OPSD) for large language models. By evaluating 16 PI setups across three agentic benchmarks and multiple student models, it finds that PI effectiveness varies with content, generation source, and model scale, with information‑dense trajectories and stronger generators yielding larger gains. The authors also propose KL‑based metrics, particularly Δd−_err, to predict PI utility before training, showing that selecting PI with this metric improves post‑training performance and remains beneficial when combined with GRPO.
arXiv Computation and Language
Sep 28
Recursive Self-Improvement via On-Policy Distillation for Reasoning
The paper introduces a recursive self-improvement framework for language models that replaces an external teacher with a frozen copy of the student, enabling dynamic co-evolution (DCE) and self-refined concise learning (SRCL). DCE allows the privileged teacher to evolve alongside the student, while SRCL trains on shorter, verified rewrites to reduce verbosity. Experiments show that the combined DCE+SRCL approach outperforms traditional on‑policy self‑distillation across multiple model sizes and math benchmarks, achieving significant accuracy gains and shorter outputs.
arXiv AI
Aug 6
Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation
arXiv:2608. 04794v1 Announce Type: new Abstract: Self-distillation (SD) has emerged as a compute-efficient alternative to reinforcement learning with verifiable rewards: a self-teacher, conditioned on privileged information (PI) about the answer such as a reference solution, supplies dense per-token supervision to a student that never sees it.
arXiv AI
Aug 11
Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation
arXiv:2608. 08176v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) improves the reasoning abilities of LLMs by internalizing privileged context into model parameters through self-distillation.
arXiv AI
Aug 19
SOD: Step-wise On-policy Distillation for Small Language Model Agents
SOD: Step-wise On-policy Distillation for Small Language Model Agents proposes a new framework that adaptively reweights distillation strength at each reasoning step based on step-level divergence. This approach mitigates cascading errors in tool-integrated reasoning by attenuating misleading teacher signals in high-divergence regions while preserving dense guidance where student and teacher align. Experiments on math, science, and code benchmarks show up to 20.86% improvement over the second-best baseline, with a 0.6B student scoring 26.13% on AIME 2025.
arXiv AI
Oct 21
Ask to Distill: Learning to Synthesize Data for On-Policy Distillation
arXiv:2610.27450v1 Announce Type: new Abstract: On-policy distillation (OPD) can succeed with surprisingly few training questions, motivating a view of questions as carriers of distributional knowled...