← Back to all news
arXiv Machine Learning By Qijia He, Zequn Li, Spencer Luo, Tianle Chen, Pradyumna Narayana, Zixian Ma, Ranjay Krishna, Lan Nie, Sugato Basu

DTDD: Divergence-Triggered Dynamic Distillation for Reliable On-Policy Supervision

Read the original on arXiv Machine Learning →

The paper introduces DTDD, a Divergence-Triggered Dynamic Distillation framework that adapts on‑policy distillation by monitoring segment‑level student‑teacher divergence and initiating teacher takeover when divergence exceeds a threshold. During takeover, the student continues to receive standard OPD supervision on the prefix while the teacher generates a recovery suffix that the student learns via behavior cloning. Experiments across long‑horizon reasoning, base initialization, and multi‑turn agents show that DTDD reduces peak gradient norms, accelerates learning from a base initialization, and improves performance on ALFWorld tasks, with segment‑averaged triggers and behavior cloning proving most effective.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Aug 18

Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuning

arXiv:2608. 16333v1 Announce Type: cross Abstract: On-policy distillation (OPD) aligns a student model with a teacher's logit distribution on student-generated trajectories.

By Changhui Sun, Lanbo Liu, Hang Lei, Tong Ling, Jiahang Xie, Zhiyong Zheng, Yujia Wang, Hao Liu, Feng Xiao, Lu Liu, Yanlong Du, Zifeng Cheng, Ziwei Jiang, Qing Gu
agentsfine-tuningefficiency
More like this →
arXiv AI
Sep 28

TISD: On-Policy Self-Distillation with Trajectory Intervention

The paper introduces TISD, a trajectory-intervention self-distillation method that forces a teacher-selected branch action and then lets the student generate the suffix, distilling the full trajectory under a privileged-context-conditioned teacher. This approach addresses a data-collection bottleneck in on‑policy self‑distillation by exposing successor contexts that the student would otherwise miss. Experiments on coding and science domains show modest but consistent improvements in average performance metrics compared to baseline methods.

By Taeckyung Lee, Rinat Amankos, Jeonghye Kim, Hyungjun Yoon, Woogyeol Jin, Sung-Ju Lee
efficiency
More like this →
arXiv Machine Learning
Sep 1

Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement

The paper investigates on‑policy distillation (OPD), showing that teacher supervision during OPD contains significant noise that grows with teacher size, yet the student policy remains largely unaffected by this noise. It finds that OPD’s gains stem mainly from suppressing low‑log‑probability tokens, a process that can be replicated without a teacher. Building on this insight, the authors propose On‑Policy Self‑Adaptation (OPSA), a supervision‑free method that uses entropy‑adaptive negative advantages to improve performance on several benchmarks, outperforming both the base model and OPD.

By Yi Ding, Ruqi Zhang
reinforcement-learningefficiencybenchmarks
More like this →
arXiv AI
Sep 25

Not Every Token Is Worth Distilling: Selective Supervision for Direct-OPD

The paper introduces Selective Supervision for Direct-OPD (S$^2$D-OPD), a refinement of Direct On-Policy Distillation that filters out states where the teacher’s policy change is minimal, as measured by the teacher‑reference Jensen‑Shannon divergence. By masking low‑divergence states and keeping only the top 10% of states per response, S$^2$D-OPD improves held‑out accuracy on AIME and HMMT benchmarks across multiple teacher‑student pairs without additional forward passes.

By Yibo Zhao, Zixuan Yang, Yunshi Lan, Xiang Li
reinforcement-learningefficiencybenchmarks
More like this →