← Back to all news
arXiv Machine Learning By Yutong Chen, Guangfu Guo, Zhichao Xu, Kunpeng Liu

DualOPSD: Adaptive Privileged Teachers for On-Policy Self-Distillation

Read the original on arXiv Machine Learning →

DualOPSD introduces an asymmetric alternating framework for on-policy self-distillation, where a privileged teacher first trains the student and then adapts to the student’s updated distribution on the same trajectory, eliminating the need for additional rollouts. This adaptive approach improves average performance on AIME 2024, AIME 2025, and HMMT 2025 benchmarks for the Qwen3-8B model, with gains that vary by model scale. Across 1.7B, 4B, and 8B scales, DualOPSD also reduces truncation effects and lowers KL divergence between teacher and student.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Sep 1

Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement

The paper investigates on‑policy distillation (OPD), showing that teacher supervision during OPD contains significant noise that grows with teacher size, yet the student policy remains largely unaffected by this noise. It finds that OPD’s gains stem mainly from suppressing low‑log‑probability tokens, a process that can be replicated without a teacher. Building on this insight, the authors propose On‑Policy Self‑Adaptation (OPSA), a supervision‑free method that uses entropy‑adaptive negative advantages to improve performance on several benchmarks, outperforming both the base model and OPD.

By Yi Ding, Ruqi Zhang
reinforcement-learningefficiencybenchmarks
More like this →
arXiv AI
22h ago

Correcting or Continuing? What On-Policy Distillation Learns from Student Errors

The paper investigates on‑policy distillation (OPD), a method that trains a student model using its own rollouts while a teacher remains frozen, to see if it can help a student improve on problems it currently fails. Experiments with Qwen‑3 and Phi‑4 models on a competition‑mathematics benchmark show that OPD actually degrades performance when training focuses on failure cases, even though training loss (KL divergence) continues to drop. However, when OPD is applied to a broader set of problems, it improves accuracy by 4.6 percentage points, indicating that OPD works well when most training data contains correct student prefixes but can reinforce errors if the training regime concentrates on failures.

By Shuo Yang, Zheyu Zhang, Mahito Sugiyama, Gjergji Kasneci
efficiency
More like this →
arXiv AI
Sep 18

RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning

RetireOPD introduces a self-retiring on‑policy distillation method for agentic reinforcement learning. It first trains a skill‑conditioned teacher with environment rewards, then jointly trains a skill‑free student with RL and OPD, allowing the student to autonomously stop using the teacher when its performance aligns with the teacher’s. Experiments on Qwen2.5 models show significant gains in ALFWorld success rates and WebShop accuracy compared to RL baselines and the teacher itself.

By Yan Yu, Zhengxi Lu, Yizhou Liu, Yichen Pan, Aozhe Wang, Qipeng Chen, Hua Yang, Wenqi Zhang, Weiming Lu, Qianglong Chen, Yongliang Shen
agentsreinforcement-learningefficiency
More like this →