Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
arXiv:2609.38025v1 Announce Type: cross Abstract: On-policy distillation (OPD) trains a student on its own generated responses using dense, token-level supervision from a stronger teacher. Vanilla OP...
arXiv:2606. 22600v2 Announce Type: replace-cross Abstract: On-Policy Distillation (OPD) improves the learning efficiency of standard reinforcement learning through dense, token-level supervision from teachers.
arXiv:2607. 04037v1 Announce Type: cross Abstract: On-policy distillation is a powerful way to transfer reasoning ability from a strong teacher to a smaller student: the student samples trajectories from its own policy, and the teacher provides dense token-level supervision on the states the student actually visits.
arXiv:2607. 28582v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) is a promising approach to improve reasoning language models, but it remains brittle in practice: making it work reliably often requires substantial engineering effort.
The paper introduces Student-Delta On-Policy Distillation (SD-OPD), a method that refines language-model reasoning by using the student’s own pretrained base as a reference for teacher feedback. Unlike traditional on‑policy distillation (OPD), SD‑OPD scales updates based on centered student‑base differences, applies them only when they align with teacher‑student feedback, and recomputes the student signal each training round. Experiments with Qwen3 students ranging from 1.7B to 8B parameters show that SD‑OPD boosts six‑task mathematics average pass@1 by 7.76 to 10.37 percentage points over OPD.
arXiv:2608. 08176v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) improves the reasoning abilities of LLMs by internalizing privileged context into model parameters through self-distillation.