arXiv Machine Learning
Unpacking the Compute Efficiency of Strong-to-Weak OPD against GRPO
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Oct 15
Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Training
The paper proposes a reward‑density principle for language‑model post‑training: sparse sequence‑level rewards are best for exploring and discovering new behaviors in a strong teacher, while dense token‑level teacher supervision is optimal for compressing that behavior into a smaller deployment model. The authors validate this rule with a four‑stage workflow—teacher RL, forward‑KL warmup, on‑policy distillation, and optional post‑bridge student RL—showing that an RL‑improved 8B teacher distilled through a dense bridge outperforms direct GRPO on a 1.7B student across math and code benchmarks. Ablation studies confirm each stage’s contribution, and the approach generalizes to Llama and Qwen models, offering a practical guideline for allocating scarce labeled data.
arXiv AI
Sep 25
Not Every Token Is Worth Distilling: Selective Supervision for Direct-OPD
The paper introduces Selective Supervision for Direct-OPD (S$^2$D-OPD), a refinement of Direct On-Policy Distillation that filters out states where the teacher’s policy change is minimal, as measured by the teacher‑reference Jensen‑Shannon divergence. By masking low‑divergence states and keeping only the top 10% of states per response, S$^2$D-OPD improves held‑out accuracy on AIME and HMMT benchmarks across multiple teacher‑student pairs without additional forward passes.
arXiv AI
Sep 4
Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation
The paper introduces Teacher-Gated On-Policy Distillation (TGOPD), a method that verifies teacher reliability at the prompt level before applying dense supervision in on-policy distillation. TGOPD uses verifier-scored teacher probes to decide whether to route a prompt to dense OPD or to a verifier-grounded alternative. Experiments on 4B and 35B models across mathematics, code, and instruction tasks show TGOPD outperforms vanilla OPD and improves teacher GPU utilization from 9.8% to 78.9% in a 4B single-domain run.
arXiv Machine Learning
Aug 14
I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimization
arXiv:2608. 12957v1 Announce Type: new Abstract: Group Relative Policy Optimization (GRPO) learns from reward differences within a rollout group, but receives no useful relative signal when every sampled response is incorrect.
arXiv AI
Aug 11
Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillation
arXiv:2608. 09826v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards yields no group-relative signal when rollout groups are uniformly correct or uniformly wrong, which account for 63.
arXiv AI
Sep 18
D$^3$-MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation
The paper introduces D$^3$-MOPD, a dynamic domain scheduling method for multi-teacher on‑policy distillation. It adapts the domain mixture during training by monitoring each domain’s reverse‑KL trajectory, thereby allocating more compute to slower‑converging domains and less to those that plateau early. Experiments on a Qwen3.6‑35B‑A3B student show that D$^3$-MOPD closes 97% of the student‑to‑teacher performance gap, matches peak performance with roughly three times fewer rollout steps, and outperforms specialist teachers on most benchmarks.