arXiv Machine Learning
RM-OPD: On-Policy Distillation for Generative Reward Models
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
2d ago
A Stronger Teacher Within: Strengthening MoE Models via On-Policy Bidirectional Distillation
The paper introduces MoEvolve, a bidirectional distillation framework that leverages increased activation budgets in Mixture-of-Experts (MoE) models to create a stronger internal teacher. By first restoring a high‑k model and then transferring its enhanced capabilities back to the original low‑k configuration, MoEvolve improves performance across various MoE models and benchmarks, achieving up to 22.2 domain‑average points and outperforming same‑budget RLVR baselines by up to 6.5 points. The resulting low‑k models often surpass their activation‑expanded teachers.
arXiv Computation and Language
Aug 28
Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms
The paper investigates three fusion paradigms—Merge, Mix RL, and multi‑teacher on‑policy distillation (MOPD)—for consolidating reinforcement learning with verifiable rewards (RLVR) across multiple domains. Experiments across model scales and a multi‑domain benchmark show that while overall performance differences are small, significant gaps can appear on specific tasks, and each method exhibits distinct training dynamics and constraints. Practical guidelines are offered: Merge for cheap fusion when experts exist, Mix RL for unified training with adjustable domain mixtures, and MOPD when preserving domain‑specific gains is paramount.
arXiv AI
Oct 9
PIVOT: Perplexity-Informed KD-to-RL Transition Scheduling for Vertical-Domain Few-Shot Distillation
PIVOT introduces a dynamic transition framework for vertical-domain few-shot distillation, routing samples between On-Policy Distillation (OPD) and GRPO-based reinforcement learning based on teacher-evaluated perplexity. Low-perplexity samples are moved to GRPO for reward-driven refinement, while high-perplexity samples remain under OPD to continue acquiring domain knowledge. Experiments on Banking77 and HWU64 demonstrate that PIVOT outperforms both continued OPD and globally synchronized OPD→GRPO baselines, achieving stronger downstream performance and more stable training dynamics.
arXiv Machine Learning
Jun 30
MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training
arXiv:2606. 30406v1 Announce Type: cross Abstract: Modern large language models (LLMs) rely on reinforcement learning during post-training to push specific capabilities, yet integrating multiple capabilities into one model remains hard.
arXiv Machine Learning
Oct 8
A Good Self-Teacher Meets the Student Where They Are: Joint On-Policy Learning and Teaching
The paper introduces Joint On-Policy Learning and Teaching (JOLT), a method that trains a single policy to act both as a privileged teacher and an unprivileged student. It derives a condition ensuring the teacher’s distillation update aligns positively with the student’s reward gradient, leading to a KL‑regularized teacher objective that matches the student’s current capabilities. Experiments show JOLT improves training efficiency and performance across various tasks, with additional gains when student rewards are incorporated.
arXiv AI
Oct 12
Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation
The paper introduces Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation (Road MOPD), a method that adds non-routed teachers as auxiliary cross-domain critics to address gradient conflicts in multi-teacher distillation. It employs token-level credit assignment based on teacher agreement and preventive gradient projection to coordinate updates and reduce conflicts. Experiments show that Road MOPD improves performance, especially in high-conflict scenarios, and mitigates degradation in disadvantaged domains.