Open-MOPD addresses the capability imbalance problem in multi-teacher on-policy distillation (M-OPD) by isolating capability integration from routing ambiguity and revealing a 35.6% headroom gap compared to a domain-routed oracle ensemble. The study identifies three key factors—sequence-length disparities, convergence drift, and reward staleness—that misallocate token-level optimization budgets, leading to severe degradation in concise tasks. The proposed Open-MOPD framework introduces token-share balancing, gap-aware dynamic budget allocation, and student reward refresh, boosting headroom recovery to 83.4% and providing an open-source, reproducible post‑training recipe and evaluation suite.
arXiv:2606. 01476v1 Announce Type: new Abstract: On-Policy Distillation (OPD) trains a student model on its own generative trajectories under dense token-level feedback from a stronger teacher, mitigating both the off-policy distribution shift of Supervised Fine-Tuning (SFT) and the sparse credit assignment of Reinforcement Learning (RL).
arXiv:2605. 07804v3 Announce Type: replace-cross Abstract: On-policy distillation (OPD) leverages dense teacher rewards to enhance reasoning models.
arXiv:2608. 03092v1 Announce Type: cross Abstract: We aim to improve model performance in multi-reward reinforcement learning training process.
arXiv:2605. 03677v2 Announce Type: replace Abstract: On-policy distillation (OPD) has recently emerged as an effective post-training paradigm for consolidating the capabilities of specialized expert models into a single student model.
arXiv:2607. 13399v1 Announce Type: cross Abstract: On-policy distillation (OPD) has become a key paradigm in LLM post-training, yet its training dynamics remain poorly understood.
arXiv:2607. 16955v1 Announce Type: cross Abstract: On-policy knowledge distillation transfers reasoning from large teachers to compact students, but existing approaches suffer three compounding failure modes: (i) cold-start collapse, where a fresh student assigns near-zero mass to teacher-preferred tokens; (ii) state-agnostic divergence scheduling, where time-only forward/reverse-KL interpolation ignores the student's coverage state; and (iii) binary reward sparsity, where pass/fail signals discard information from partially correct traces.
arXiv:2608. 11698v1 Announce Type: cross Abstract: On-policy distillation (OPD) trains a student on its own trajectories under dense token-level supervision from a teacher.
arXiv:2608. 12146v1 Announce Type: cross Abstract: Training Mixture-of-Experts (MoE) models for reinforcement learning (RL) couples two load-balancing problems: sequence composition determines dense attention work in each data-parallel microbatch, while token routing determines sparse expert work on expert-parallel ranks.
arXiv:2606. 29476v1 Announce Type: cross Abstract: Self-distilled agentic reinforcement learning augments trajectory-level reward with a token-level distillation loss, using as its teacher the same policy conditioned on privileged context.
arXiv:2606. 21994v2 Announce Type: replace Abstract: On-policy distillation (OPD) improves reasoning models by applying dense teacher supervision on student-sampled trajectories.
arXiv:2606. 26671v1 Announce Type: new Abstract: Post-training alignment determines the reasoning and human preference following capabilities of large language models, yet most existing works withhold detailed data construction, filtering rules and training recipes, which hinders community reproducibility and lightweight model optimization.
arXiv:2608. 08176v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) improves the reasoning abilities of LLMs by internalizing privileged context into model parameters through self-distillation.