The paper introduces uncertainty‑calibrated Multi‑Teacher On‑Policy Distillation (MOPD) to better preserve general language model capabilities while specializing to specific domains. By employing dual‑temperature sampling, positive‑advantage‑density filtering, and centered log‑likelihood filtering, the method selects more informative trajectories and token updates, leading to significant improvements in general‑capability performance on role‑playing and medical‑domain tasks without sacrificing domain performance.
arXiv:2610.02324v1 Announce Type: cross Abstract: Foundation multimodal large language models are designed to support a broad spectrum of capabilities across diverse domains. Multi-teacher on-policy...
The paper introduces Counteraction-Aware Multi-Teacher On-Policy Distillation (CaMOPD), a method designed to recover general capabilities in large language models while preserving domain-specific behavior. CaMOPD tackles two failure modes of standard Multi-Teacher On-Policy Distillation—conflicting recovery and preservation gradients, and weak correction signals—by using decoupled alternating training and selecting samples with large teacher‑student log‑probability gaps. Experiments on role‑play dialogue and medical reasoning QA show that CaMOPD outperforms baselines in general capability recovery while maintaining domain specialization, and gradient coherence analyses confirm more coherent correction signals.
arXiv:2610.20174v1 Announce Type: new Abstract: Reinforcement Learning on Mixed-domain preference data (Mix-RL) has become the mainstream approach for training general-purpose Generative Reward Model...
The paper investigates on‑policy distillation (OPD) for large language models, showing that OPD learns not only which tokens to consider but also where to keep reasoning paths open. It introduces the entropy shape—a sequence of next‑token entropies—to capture this "where to branch" aspect and proposes the Entropy Shape Correlation (ESC) metric to measure student‑teacher agreement. By tracking ESC during training and adjusting a teacher’s entropy while preserving token preferences, the authors demonstrate that controlling where to branch causally drives OPD gains, and they establish a teacher‑selection protocol where pre‑OPD ESC predicts the best teacher more accurately than existing methods.
arXiv:2606. 30626v1 Announce Type: new Abstract: On-policy distillation (OPD) offers superior capacity transfer by supervising student-sampled trajectories with dense token-level signals.
arXiv:2605. 03677v2 Announce Type: replace Abstract: On-policy distillation (OPD) has recently emerged as an effective post-training paradigm for consolidating the capabilities of specialized expert models into a single student model.
arXiv:2604. 10688v2 Announce Type: replace-cross Abstract: On-policy reinforcement learning has become the dominant paradigm for reasoning alignment in large language models, yet its sparse, outcome-level rewards make token-level credit assignment notoriously difficult.
Reverse-KL Policy Distillation explores the statistical guarantees of training language models using reverse Kullback–Leibler divergence. The study compares online and offline distillation, showing that online learning requires more data when only the probability of the student’s chosen action is observed, while offline learning achieves the same minimax rate regardless of whether the teacher’s full distribution or only the probability of the teacher‑generated action is revealed. The work also demonstrates that standard online objectives can fail and introduces trajectory‑loss, importance‑ratio truncation, and policy‑space learning to attain optimal guarantees.
The paper investigates the statistical properties of Reverse‑Kullback‑Leibler (KL) policy distillation, focusing on how different feedback settings affect sample complexity. It shows that when the teacher reveals its full action distribution, online distillation achieves minimax rates, whereas revealing only the probability of the student‑generated action requires substantially more data. In contrast, offline distillation attains the same minimax rate regardless of whether the teacher’s full distribution or only the probability of the teacher‑generated action is revealed, because teacher‑generated actions expose the teacher’s preferences directly.
The paper studies how different configurations of privileged information (PI) affect on‑policy self‑distillation (OPSD) for large language models. By evaluating 16 PI setups across three agentic benchmarks and multiple student models, it finds that PI effectiveness varies with content, generation source, and model scale, with information‑dense trajectories and stronger generators yielding larger gains. The authors also propose KL‑based metrics, particularly Δd−_err, to predict PI utility before training, showing that selecting PI with this metric improves post‑training performance and remains beneficial when combined with GRPO.
On-policy distillation (OPD) transfers teacher capabilities by supervising trajectories sampled from the student's own policy, yet its generalization behavior remains poorly understood, as most studies evaluate OPD on a single domain and on benchmarks close to the training data. We present a controlled study that varies one generalization factor at a time, from in-domain distribution shifts to cross-domain transfer and the multi-teacher setting.
MOPD‑Router rethinks teacher routing in multi‑teacher on‑policy distillation by routing supervision over the full teacher pool at each token, eliminating the need for prompt‑level domain labels or a separate routing model. The framework offers a plug‑in interface for various metrics, and introduces ExpertAlign, which scores teachers based on how well their corrections reflect their specialized post‑training knowledge. Experiments on both unlabeled and domain‑labeled mixtures show that ExpertAlign outperforms existing methods, improving overall scores by up to 12.3% on unlabeled data and 7.8% on domain‑labeled data. whyItMatters":"Token‑level routing enables the use of complementary supervision across domains without relying on domain labels, leading to significant performance gains in multi‑teacher distillation settings."