arXiv AI
Composing What Each Teacher Learned: Delta Distillation for Multi-Teacher On-Policy Learning
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Oct 8
Composing What Each Teacher Learned: Multi-Teacher On-Policy Distillation through Teacher-Relative Shifts
The paper introduces Δ‑MOPD, a multi‑teacher on‑policy distillation method that transfers each teacher’s logit shift relative to its base policy, re‑anchored at the student’s frozen initialization. Δ‑MOPD is evaluated in two settings: common‑domain composition, where multiple teachers score a single student rollout, and routed‑domain distillation, where prompts are assigned to specialist teachers. Across experiments, Δ‑MOPD outperforms traditional endpoint supervision, especially when teacher signals are combined at a state, achieving higher accuracy and reducing performance gaps in phased routing scenarios.
arXiv Machine Learning
Sep 15
Know When to Stop, Where to Restart: Accelerating Multi-Turn Agentic On-Policy Distillation
arXiv:2609.14636v1 Announce Type: new Abstract: On-policy distillation (OPD) has become a standard approach for transferring capabilities from large teachers to compact students. Its cost, however, i...
arXiv AI
Sep 28
MOPD-Router: Rethinking Teacher Routing in Multi-Teacher On-Policy Distillation
MOPD‑Router rethinks teacher routing in multi‑teacher on‑policy distillation by routing supervision over the full teacher pool at each token, eliminating the need for prompt‑level domain labels or a separate routing model. The framework offers a plug‑in interface for various metrics, and introduces ExpertAlign, which scores teachers based on how well their corrections reflect their specialized post‑training knowledge. Experiments on both unlabeled and domain‑labeled mixtures show that ExpertAlign outperforms existing methods, improving overall scores by up to 12.3% on unlabeled data and 7.8% on domain‑labeled data. whyItMatters":"Token‑level routing enables the use of complementary supervision across domains without relying on domain labels, leading to significant performance gains in multi‑teacher distillation settings."
arXiv AI
Sep 25
Not Every Token Is Worth Distilling: Selective Supervision for Direct-OPD
The paper introduces Selective Supervision for Direct-OPD (S$^2$D-OPD), a refinement of Direct On-Policy Distillation that filters out states where the teacher’s policy change is minimal, as measured by the teacher‑reference Jensen‑Shannon divergence. By masking low‑divergence states and keeping only the top 10% of states per response, S$^2$D-OPD improves held‑out accuracy on AIME and HMMT benchmarks across multiple teacher‑student pairs without additional forward passes.
arXiv AI
Oct 15
Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Training
The paper proposes a reward‑density principle for language‑model post‑training: sparse sequence‑level rewards are best for exploring and discovering new behaviors in a strong teacher, while dense token‑level teacher supervision is optimal for compressing that behavior into a smaller deployment model. The authors validate this rule with a four‑stage workflow—teacher RL, forward‑KL warmup, on‑policy distillation, and optional post‑bridge student RL—showing that an RL‑improved 8B teacher distilled through a dense bridge outperforms direct GRPO on a 1.7B student across math and code benchmarks. Ablation studies confirm each stage’s contribution, and the approach generalizes to Llama and Qwen models, offering a practical guideline for allocating scarce labeled data.
arXiv AI
Aug 11
WDL-OPD: Weak-Driven On-Policy Distillation via Mixture-Constrained Co-Training
arXiv:2608. 09447v1 Announce Type: cross Abstract: On-policy distillation (OPD) aligns a student with a teacher on trajectories sampled from the student itself, reducing the train-test state mismatch of offline distillation.